Part 6 put required_scopes=[\"mcp:invoke\"] on the JWT\nverifier and called the server authorised. It is, in the sense that an unauthenticated caller\ngets nothing. It is not, in the sense that matters: mcp:invoke opens every tool on the server.\nThe same token that lets an agent run find_customer lets it run issue_refund.
That is the whole gap. A read-only agent and a refund-issuing agent hold identical credentials,\nand the only thing standing between \"look up Maria\" and \"refund invoice 2\" is that nobody asked.
\nThis post closes it, badly first and then properly, because the badly is what most people ship\nand the difference only shows up in what the client can do about a refusal.
\nBoth versions refuse the same call ā a read-only token asking for issue_refund. They differ in\nwhere the scope is checked, and that one choice decides what the client gets back:
Parts 5 and 6, specifically:
\nagent-tools provider\n(part 5)~/mcp-auth/token.sh, which takes a scope string and returns an access tokenPart 6's server stays on :8770 throughout. The two versions below run on :8771 and :8772\nso you can compare all three without stopping anything.
Scopes are Property Mappings in Authentik. Customization ā Property Mappings ā New Property\nMapping ā Scope Mapping, twice:
\n| Name | Scope name | Description |
|---|---|---|
office-read | office:read | Look up customers and invoices |
office-refund | office:refund | Issue refunds |
![\"Two](\"https://development-wec.wiline.com/docs/assets/images/h10-scope-mappings-c80d52648d4347ac31467ecd7bfbc924.png\")
Creating them is not enough. A provider will only issue a scope it has been given, so open\nApplications ā Providers ā agent-tools ā Edit and move both into Selected Scopes.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/h10-provider-scopes-845343ec44444c00ad1c5931086ff83c.png\")
Note the line under the picker: \"Select which scopes can be used by the client. The client still\nhas to specify the scope to access the data.\" Both halves matter. Selecting a scope here does not\nput it in every token ā it permits the client to ask. A client that asks for nothing gets nothing,\nwhich is why every token.sh call below passes an explicit scope string.
Confirm the token actually carries what you asked for:
\n~/mcp-auth/token.sh \"mcp:invoke office:read\" \\
| cut -d. -f2 | base64 -d 2>/dev/null | jq .scope
\"mcp:invoke office:read\"
If that comes back without office:read, the scope is not on the provider ā fix that before\nwriting any server code, or you will spend an hour debugging enforcement that is working\ncorrectly on a token that was never scoped.
Keep required_scopes=[\"mcp:invoke\"] on the verifier as the price of admission, then ask per\ntool whether the caller holds what that operation needs. FastMCP exposes the verified token\nthrough get_access_token(), so a decorator can read the claims the verifier already checked:
from fastmcp.exceptions import ToolError
from fastmcp.server.dependencies import get_access_token
def requires(scope: str):
def decorate(fn):
@functools.wraps(fn)
async def wrapper(*args, **kwargs):
token = get_access_token()
held = set(getattr(token, \"scopes\", None) or [])
if scope not in held:
raise ToolError(
f'insufficient_scope: this tool requires \"{scope}\"; '
f'token carries {sorted(held) or \"nothing\"}'
)
return await fn(*args, **kwargs)
return wrapper
return decorate
Then one line per tool:
\n@mcp.tool
@requires(\"office:read\")
async def find_customer(query: str) -> list[dict]: ...
@mcp.tool
@requires(\"office:refund\")
async def issue_refund(invoice_id: int) -> str: ...
Run it on :8771 and drive both tools with both tokens:
![\"Two](\"https://development-wec.wiline.com/docs/assets/images/h10-per-tool-enforcement-36365bd959bc324dd447cfdbe681525c.png\")
--- token: mcp:invoke + office:read ---
find_customer : ALLOWED -> [{'id': 1, 'name': 'Maria Alvarez', ...}]
issue_refund : REFUSED -> insufficient_scope: this tool requires \"office:refund\"
--- token: mcp:invoke + office:refund ---
find_customer : REFUSED -> insufficient_scope: this tool requires \"office:read\"
issue_refund : ALLOWED -> Refunded 80.00 on invoice 2.
That is real enforcement. The refund token cannot read, the read token cannot refund, and the\nrefusal names the missing scope. For a lot of deployments this is where you stop.
\nWatch the HTTP layer rather than the client library.
\nTOK=$(~/mcp-auth/token.sh \"mcp:invoke office:read\")
curl -sS -i -X POST http://127.0.0.1:8771/mcp \\
-H \"Authorization: Bearer $TOK\" \\
-H \"Content-Type: application/json\" \\
-H \"Accept: application/json, text/event-stream\" \\
-d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"issue_refund\",\"arguments\":{\"invoice_id\":2}}}'
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h10-200-no-challenge-ec64c16f2444a9ed7bcacaed0d917f3e.png\")
HTTP/1.1 200 OK
content-type: text/event-stream
data: {\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"content\":[{\"text\":\"insufficient_scope: this tool
requires \\\"office:refund\\\"; token carries ['mcp:invoke', 'office:read']\",\"type\":\"text\"},
\"isError\":true}}
HTTP 200. The request succeeded; the tool declined. That is correct JSON-RPC semantics and\nit is useless to an OAuth client.
\nAn OAuth client that wants to step up ā go back to the authorization server and ask for\noffice:refund ā is watching for a 401 or 403 with a WWW-Authenticate header telling it\nwhat to request. It does not parse English out of a tool result. So the refusal is legible to a\nhuman reading logs and invisible to the machinery designed to handle exactly this case.
It also arrives late. The tool body runs after routing, after session setup, after the server has\ncommitted to a successful response. By then there is no status line left to change.
\nGetting a 403 means checking scopes before the JSON-RPC layer answers, which means ASGI\nmiddleware wrapping the FastMCP app:
class ScopeChallengeMiddleware:
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope[\"type\"] != \"http\" or scope[\"method\"] != \"POST\":
return await self.app(scope, receive, send)
# Buffer the body so we can inspect it and still pass it on.
chunks, more = [], True
while more:
msg = await receive()
chunks.append(msg.get(\"body\", b\"\"))
more = msg.get(\"more_body\", False)
body = b\"\".join(chunks)
needed = None
try:
rpc = json.loads(body)
if rpc.get(\"method\") == \"tools/call\":
needed = TOOL_SCOPES.get((rpc.get(\"params\") or {}).get(\"name\"))
except Exception:
pass
if needed and needed not in token_scopes(dict(scope.get(\"headers\") or [])):
challenge = (
f'Bearer error=\"insufficient_scope\", scope=\"{needed}\", '
f'resource_metadata=\"{RESOURCE_METADATA}\", '
f'error_description=\"This operation requires the {needed} scope\"'
)
await send({\"type\": \"http.response.start\", \"status\": 403,
\"headers\": [(b\"content-type\", b\"application/json\"),
(b\"www-authenticate\", challenge.encode())]})
await send({\"type\": \"http.response.body\",
\"body\": json.dumps({\"error\": \"insufficient_scope\",
\"scope\": needed}).encode()})
return
# Replay the buffered body downstream.
replayed = False
async def replay():
nonlocal replayed
if not replayed:
replayed = True
return {\"type\": \"http.request\", \"body\": body, \"more_body\": False}
return await receive()
await self.app(scope, replay, send)
app = ScopeChallengeMiddleware(mcp.http_app(stateless_http=True))
Same request, against :8772:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h10-403-challenge-2a8d3ff52be96195ff042471605c0c4e.png\")
HTTP/1.1 403 Forbidden
www-authenticate: Bearer error=\"insufficient_scope\", scope=\"office:refund\",
resource_metadata=\"http://127.0.0.1:8772/.well-known/oauth-protected-resource\",
error_description=\"This operation requires the office:refund scope\"
Now a client has something to act on: the status says refused, scope= says what to ask for, and\nresource_metadata says where to look up how. That is step-up authorization as a protocol rather\nthan as a log message.
The middleware works. It is also worse code than the decorator, in three specific ways, and\npretending otherwise would be dishonest.
\nIt does not know what a tool is. ASGI middleware sees bytes and headers. To find out which\ntool is being called it parses the JSON-RPC envelope itself and looks the name up in a table it\nhas to maintain:
\nTOOL_SCOPES = {
\"find_customer\": \"office:read\",
\"open_invoices\": \"office:read\",
\"issue_refund\": \"office:refund\",
}
That table is a second source of truth. Add a tool and forget the entry and it is unprotected ā\nsilently, because the middleware just passes through anything it does not recognise. The\ndecorator could not have that bug: the requirement sat on the function.
\nIt decodes the token unverified. The middleware runs upstream of the verifier, so the\nverified claims do not exist yet. It splits the JWT and base64-decodes the payload without\nchecking the signature:
\npayload = auth.split(None, 1)[1].split(\".\")[1]
claims = json.loads(base64.urlsafe_b64decode(payload + \"=\" * (-len(payload) % 4)))
This is not the hole it looks like ā the verifier still runs downstream and still rejects a\nforged token, so nothing reaches a tool on a bad signature. But the scope decision is made on\nunauthenticated bytes, and the only reason that is survivable is the second check behind it. It\nis a wart, not a vulnerability, and it is the kind of thing worth writing down before someone\nlater removes the \"redundant\" verifier.
\nIt buffers every request body. To read the envelope and still pass it downstream, the\nmiddleware drains receive() into memory and replays it. Fine for JSON-RPC calls; think harder\nbefore putting this in front of large uploads.
insufficient_scope on a tool you did grantāCheck the token, not the server:
\n~/mcp-auth/token.sh \"mcp:invoke office:refund\" | cut -d. -f2 | base64 -d 2>/dev/null | jq .scope
If office:refund is missing, the scope exists as a Property Mapping but was never moved into\nthe provider's Selected Scopes. Authentik silently drops scopes a client is not permitted to\nrequest rather than erroring, so the token comes back valid and short.
It only inspects POST. MCP clients open a GET for the event stream first, and that request\ncarries no JSON-RPC envelope ā if you are watching the wrong request you will conclude the\nmiddleware is dead. Confirm with the curl above, which is a single POST.
TOOL_SCOPES is keyed by the tool's registered name, which is the function name, not the\ndecorated label. Rename a function and the table stops matching. The pass-through case is the\ndangerous direction (unprotected), but a typo in the table produces this one.
The challenge lives in the WWW-Authenticate header. Clients that only log response bodies\nwill show you {\"error\":\"insufficient_scope\"} and nothing about which scope. Use curl -i.
It bought genuine per-tool authorisation: two tokens that differ by one scope, each able to\nrun exactly one of two tools, proven at the HTTP layer rather than asserted. And with the\nmiddleware, a refusal a client can programmatically recover from.
\nIt did not buy a clean design. Both implementations are compromises pointing opposite ways:
\ntool decorator (:8771) | ASGI middleware (:8772) | |
|---|---|---|
| Scope requirement lives | on the function | in a separate table |
| Reads verified claims | yes | no ā decodes unverified, verifier runs after |
| Refusal | HTTP 200, isError: true | HTTP 403 + WWW-Authenticate |
| Client can step up | no | yes |
| New tool unprotected by default | no | yes |
The honest summary is that the correct protocol behaviour requires leaving the abstraction the\nframework gives you, and the ergonomic version cannot produce it. If your clients do not\nimplement step-up ā and most agent clients today do not ā the decorator is the better trade.\nIf they do, you pay for it with a table you must remember to update.
\nIt also did not buy authorisation that survives the tool doing something else. issue_refund\nis gated on office:refund; nothing stops a future find_customer from being edited to write.\nScopes gate entry, not behaviour ā which is the same boundary\npart 4 drew around policy-in-the-tool.
The obvious remaining gap is that both versions trust the token's scope list and nothing else.\nNeither asks who the caller is or what they are acting on ā a token with office:refund\nrefunds any invoice, for any customer, for any amount. That is object-level authorisation, and\nit does not live in OAuth scopes at all.
WWW-Authenticate challengeEvery agent tutorial in this series so far has given a model tools ā functions you wrote,\nwith arguments you defined. This post is about the other thing agents do, which is write code\nand then run it.
\nThat is a different risk, and the difference is worth being precise about. A tool call is the\nmodel choosing from a menu you control. Executing generated code is the model handing you\nsomething nobody has ever read, which you then run on your machine.
\n\nOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Ubuntu 22.04, kernel 5.15, systemd 249.\nDocker rootless 29.8.1 from Part 8, running\nalongside the system daemon. Test image is python:3.12-alpine. The host had 20 containers\nrunning throughout.
Every number and error message below is from that run.
The industry has a name for this. OWASP's\nLLM05:2025 Improper Output Handling\nlists it as a vulnerability example, in plain words:
\n\n\nLLM output is entered directly into a system shell or similar function such as exec or eval,\nresulting in remote code execution.
\n
And its advice on how to think about the model is exactly right:
\n\n\nTreat the model as any other user, adopting a zero-trust approach
\n
Here is what makes that page worth reading closely, though. It tells you to treat model output\nas untrusted, it names code execution as the consequence, and then its prevention section talks\nentirely about validating text ā output encoding, parameterised queries, content security\npolicies, monitoring.
\nSearch that page for sandbox, container, least privilege or isolation and you\nget zero hits. Not few. None.
\nThat is not a criticism of OWASP; the risk is about output handling, and validation is the right\nanswer for output that becomes HTML or SQL. But when the output is a program you are about to\nrun, validation stops being possible. You cannot regex your way to knowing whether a hundred\nlines of Python are safe.
\nSo the guidance names the problem and stops one layer above the fix. This post is that layer.
\nThe naive implementation is what most people ship first, and it looks responsible ā the code\nruns in a container, after all:
\ndocker -c rootless run --rm python:3.12-alpine python -c \"<the model's code>\"
Let's ask that container what it can reach.
\ndocker -c rootless run --rm python:3.12-alpine python -c \"
import os, urllib.request
print('uid inside :', os.getuid())
print('can see host / :', sorted(os.listdir('/'))[:10])
try:
r = urllib.request.urlopen('https://api.github.com', timeout=5)
print('internet : REACHED', r.status)
except Exception as e:
print('internet : blocked', type(e).__name__)
\"
docker -c rootless run --rm python:3.12-alpine sh -c '
echo \"memory limit : $(cat /sys/fs/cgroup/memory.max)\"
echo \"pids limit : $(cat /sys/fs/cgroup/pids.max)\"
echo \"cpus visible : $(nproc)\"
echo \"writable fs : $(touch /proc-test 2>/dev/null && echo yes || echo no)\"
'
![\"A](\"https://development-wec.wiline.com/docs/assets/images/h9-naive-sandbox-06b5b3907235d86115ef158babda8df0.png\")
internet : REACHED 200
memory limit : max
pids limit : 19136
cpus visible : 8
writable fs : yes
Read that as a list of what generated code is allowed to do on your machine:
\nmax means no limit. On this host that is 15 GB, and the kernel\nwill start killing other things to provide it.Part 8 moved the daemon off root, so this code can no longer take over the host. That was\nworth doing and it does not help here at all. None of the five items above involves being root.
\nDocker has flags for every one of these. They are unglamorous and they are the entire fix:
\ndocker -c rootless run --rm \\
--network none --memory 256m --pids-limit 64 --cpus 0.5 \\
--read-only --tmpfs /tmp:size=16m \\
--cap-drop=ALL --security-opt no-new-privileges \\
python:3.12-alpine <command>
In plain terms:
\n| Flag | What it stops |
|---|---|
--network none | No network at all ā nothing can be sent anywhere |
--memory 256m | Uses more than 256 MB and the kernel kills it, not your other services |
--pids-limit 64 | A fork bomb hits 64 and stops |
--cpus 0.5 | An infinite loop gets half a core |
--read-only | Nothing can be written to the container's filesystem |
--tmpfs /tmp:size=16m | Except a small scratch area, capped, in memory |
--cap-drop=ALL | No kernel privileges ā no mounting, no raw sockets, nothing |
--security-opt no-new-privileges | Nothing inside can gain more privileges than it started with |
Run it, and it fails before the container even starts:
\n![\"Docker](\"https://development-wec.wiline.com/docs/assets/images/h9-cpus-refused-0a8c67fa115c08d21d53054814b86f30.png\")
docker: Error response from daemon: NanoCPUs can not be set, as your kernel does not
support CPU CFS scheduler or the cgroup is not mounted
Hold onto that message, because it is misleading. The kernel supports CPU limits perfectly\nwell ā this host runs 20 other containers with them available. We will come back to what is\nactually wrong.
\nDrop --cpus and everything else applies:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h9-hardened-holds-e5ed1f26eadb61b1c097d1438eae1520.png\")
memory limit : 268435456 <- exactly 256 MiB
pids limit : 64
cpus visible : 8
writable fs : no
tmp writable : yes
internet : blocked URLError
Five of six controls working. And cpus visible: 8, because we were not allowed to set one.
A missing CPU limit sounds mild next to \"unlimited memory\". It is not, because an infinite loop\nis the single most likely thing generated code does by accident.
\nThis runs eight busy loops inside the otherwise fully locked-down sandbox, for eight seconds:
\ndocker -c rootless run -d --name burn \\
--network none --memory 256m --pids-limit 64 \\
--read-only --tmpfs /tmp:size=16m \\
--cap-drop=ALL --security-opt no-new-privileges \\
python:3.12-alpine sh -c 'for i in $(seq 8); do (while :; do :; done) & done; sleep 8'
sleep 3
docker -c rootless stats --no-stream burn
![\"Docker](\"https://development-wec.wiline.com/docs/assets/images/h9-burn-605-a829703726df3283acc0afa12d391696.png\")
NAME CPU % MEM USAGE / LIMIT NET I/O PIDS
burn 605.19% 1.23MiB / 256MiB 0B / 0B 9
605% of a CPU. Six of the host's eight cores, taken by code inside a sandbox where memory is\ncapped at 256 MB (it used 1.23), processes are capped at 64 (it used 9), and the network is\ncompletely blocked.
\nEvery control we set is working. The one we could not set is the one being abused.
\nThe error message blamed the kernel. The kernel is fine. The real answer is one file.
\nWhen Docker runs rootless, it does not get to manage resources directly ā it can only use what\nthe system has handed to your user account. On modern Linux that handover is called cgroup\ndelegation, and systemd decides what gets delegated.
\ncat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
memory pids
Two things. That is the whole explanation:
\n\nThe daemon knew all along. It said so at startup, ten times, in lines nobody reads:
\njournalctl --user -u docker.service | grep -i \"no .* support\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h9-delegation-root-cause-801bd67d214cce22c58da71540cddebb.png\")
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.max (rbps) support
...
This also explains something Part 8 reported as a\nplain limitation of rootless mode: disk throughput limits not working. Same cause. Not a\nproperty of rootless ā a property of what systemd delegated.
\nNot in Docker's rootless documentation. I searched the page: zero occurrences of\n\"delegate\". The guidance lives upstream, on the\nrootless containers site, which\nstates it directly:
\n\n\nBy default, a non-root user can only get memory controller and pids controller to be delegated.
\n
One drop-in file telling systemd to hand over more:
\nsudo mkdir -p /etc/systemd/system/user@.service.d
cat <<EOF | sudo tee /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=cpu cpuset io memory pids
EOF
sudo systemctl daemon-reload
The same page warns:
\n\n\nAfter changing the systemd configuration, you need to re-login or reboot the host. Rebooting\nthe host is recommended.
\n
On systemd 249 that turned out not to be needed. After daemon-reload alone:
cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
cpuset cpu io memory pids
The daemon still needs restarting, because it reads its capabilities once at startup. Restart\nonly the rootless daemon ā not your whole user session, which on this host would also have\ntaken down an unrelated service:
\nsystemctl --user restart docker.service
journalctl --user -u docker.service --since \"-1min\" | grep -ci \"no cpu\\|no io\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h9-delegate-fix-1f7414c6361fa1fc13be57138940d1f5.png\")
0
Zero warnings, where there were ten.
\nIdentical command to before. One flag added:
\ndocker -c rootless run -d --name burn2 \\
--network none --memory 256m --pids-limit 64 --cpus 0.5 \\
--read-only --tmpfs /tmp:size=16m \\
--cap-drop=ALL --security-opt no-new-privileges \\
python:3.12-alpine sh -c 'for i in $(seq 8); do (while :; do :; done) & done; sleep 8'
sleep 3
docker -c rootless stats --no-stream burn2
![\"Docker](\"https://development-wec.wiline.com/docs/assets/images/h9-burn-capped-7dff3d60a38ee6b89cb8667301faea0f.png\")
NAME CPU % MEM USAGE / LIMIT NET I/O PIDS
burn2 52.82% 1.242MiB / 256MiB 0B / 0B 9
605.19% ā 52.82%. Same eight infinite loops, same image, same everything. Half a core, as\nasked.
\nA container is not a sandbox until you say what it may use. The default is generous in every\ndimension that matters, and nothing warns you, because an unrestricted container is a completely\nnormal thing to run.
\nCheck your limits are applied, rather than assuming. The whole of this post exists because a\nflag that was silently unavailable looked identical to a flag that was working. docker stats\nunder a deliberate load takes a minute and tells you the truth.
Error messages point at the wrong layer. \"your kernel does not support CPU CFS scheduler\"\ndescribed a kernel that supports it fine. The real cause was a systemd default, two layers away,\ndocumented on a different website.
\nAnd the honest limit: this sandbox contains resources. It does not make the code safe. It\ncannot tell a useful script from a hostile one, and a program that does something harmful within\n256 MB, half a core and no network will run exactly as written. Containment buys you a blast\nradius, not a judgement.
\nsystemd.resource-control(5) ā Delegate=Here is the part nobody says out loud when they tell you to add yourself to the docker group\nso you can stop typing sudo.
That group is root. Not \"close to root\", not \"root for Docker things\". If you can run a\ncontainer, you can read, change or delete any file on the machine ā including the password\nfile, including other people's home directories, including the files the administrator\ndeliberately kept away from you.
\nNo exploit required. It is one command, and it takes about four seconds.
\n\nOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Ubuntu 22.04, kernel 5.15. System daemon is\nUbuntu's docker.io 29.1.3; the rootless daemon installed here is 29.8.1 with\nrootlesskit 3.1.0. Test image is alpine:3.20. The host was running 20 containers under the\nsystem daemon throughout, and none of them stopped.
Every command output and timing below is from that run.
Run this as a normal user. There is no sudo anywhere in it.
docker run --rm -v /:/host alpine:3.20 \\
sh -c 'touch /host/root/rootless-demo && echo WROTE as uid $(id -u)'
sudo ls -l /root/rootless-demo
![\"An](\"https://development-wec.wiline.com/docs/assets/images/h8-docker-group-is-root-8007f44ffccf255e2de1e18d9bf1ca00.png\")
WROTE as uid 0
-rw-r--r-- 1 root root 0 Sep 16 20:58 /root/rootless-demo
A user who cannot so much as list /root just created a file in it, owned by root.
Nothing malfunctioned. That is the design. The daemon runs as root, you asked it to mount the\nwhole filesystem, and it did ā because a root process is allowed to. The container's \"root\" is\nthe host's root. They are the same thing.
\nDocker documents this plainly, and it is worth reading their wording rather than mine. From\nDocker daemon attack surface:
\n\n\nonly trusted users should be allowed to control your Docker daemon
\n
and, on the same page:
\n\n\nDocker allows you to share a directory between the Docker host and a guest container; and it\nallows you to do so without limiting the access rights of the container.
\n
That is not a warning about a flaw. It is a description of the feature we just used.
\nClean it up before moving on:
\nsudo rm -f /root/rootless-demo
An AI agent that executes code it generated is, in security terms, remote code execution that\nyou installed on purpose. That is not a criticism of agents ā it is what they are for.
\nSo the question stops being \"could an attacker run code here?\" ā something already does,\nconstantly ā and becomes \"what can that code reach?\"
\nIf it can reach the Docker socket, the answer is everything, via the command above. Your API\nkeys. Your SSH keys. The database volume. All of it, regardless of how carefully the agent's\nown container was configured.
\nPart 2 found two of six Langfuse containers\nrunning as root and fixed them with user: \"999:999\". That was worth doing and it is still\nworth doing.
But it changes who the process is inside the container. The daemon that starts it is still\nroot, and the socket is still a door to the whole machine. Part 2 hardened the passenger. This\npost goes after the driver.
\nThe daemon stops running as root and runs as you.
\nContainers still think they have root ā software inside them keeps working normally. But that\n\"root\" is now translated, by the kernel, into an ordinary unprivileged account that owns\nnothing important. Escape the container and you are not the administrator. You are ubuntu.
The translation uses a block of user IDs the system set aside for you. Here is the whole idea\nin one picture ā the same container, under each daemon:
\n\nThe left half is why the first command worked. The right half is what we are about to build,\nand the arithmetic in it is something you can verify yourself in a minute.
\nThis translation is a Linux kernel feature called a\nuser namespace, not a Docker\ninvention ā Docker is just asking the kernel for one. The block of IDs it maps into is declared\nin /etc/subuid:
grep '^ubuntu:' /etc/subuid /etc/subgid
/etc/subuid:ubuntu:100000:65536
/etc/subgid:ubuntu:100000:65536
65,536 IDs starting at 100000, yours alone. Hold onto that number ā the mapping becomes\ncheckable arithmetic in a moment.
\nThe documentation assumes a setup this machine does not have. Here is each wall in the order\nyou hit it.
\nEvery guide opens with the same line. On Ubuntu's Docker:
\nsudo apt-get install -y docker-ce-rootless-extras
dpkg -L docker.io | grep -iE 'rootless|rootlesskit'
![\"apt](\"https://development-wec.wiline.com/docs/assets/images/h8-no-rootless-package-4540e2d1cdbee668a23f37364afdc535.png\")
E: Unable to locate package docker-ce-rootless-extras
That package belongs to Docker's own repository. This host runs Ubuntu's docker.io, and the\nsecond command returns nothing at all ā the distribution package ships no rootless tooling\nwhatsoever.
That matters because it forces a choice. You can replace docker.io with Docker CE, which\nmeans swapping out the daemon currently running your containers. Or you can install a second,\nuser-owned daemon beside the first. With 20 services running, the second option is the only\nsane one, and it is what the rest of this post does.
What you do need from apt is the ID-mapping helpers:
\nsudo apt-get install -y uidmap
On Ubuntu 22.04 this may trigger a \"Daemons using outdated libraries\" prompt with\ncontainerd.service and your database pre-selected. uidmap does not need any of them\nrestarted. Deselect them, or cancel the prompt ā the install still completes.
curl -fsSL https://get.docker.com/rootless | sh
# Installing stable version 29.8.1
Aborting because rootful Docker is running and accessible.
Set FORCE_ROOTLESS_INSTALL=1 to ignore.
The guard assumes you are migrating ā that a running root daemon means you are about to\nmake a mess. Running both side by side on purpose is never mentioned, and the way to proceed\nappears only inside the error text.
\nNote the version it names: 29.8.1, while the system daemon is 29.1.3. You will end up\nwith two different Docker versions on one host. That is expected here, and worth knowing.
\ncurl -fsSL https://get.docker.com/rootless | FORCE_ROOTLESS_INSTALL=1 sh
Read the last lines of the installer output:
\n[INFO] Creating CLI context \"rootless\"
[INFO] Using CLI context \"rootless\"
Current context is now \"rootless\"
Now ask Docker what is running:
\ndocker ps
docker context ls
docker -c default ps -q | wc -l
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/h8-context-hijack-4d422c845ec7d88ee2b2f5a5ffbed4af.png\")
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
NAME DESCRIPTION DOCKER ENDPOINT ERROR
default Current DOCKER_HOST based configuration unix:///var/run/docker.sock
rootless * Rootless mode unix:///run/user/1000/docker.sock
20
An empty list, while twenty containers run perfectly well. Nothing broke and nothing warned\nyou ā the CLI is answering a different question than the one you think you asked, because the\ninstaller repointed it on the way out.
\nPut it back, and opt in per command instead:
\ndocker context use default # everyday work, unchanged
docker -c rootless ps # explicit when you want the new one
Start the daemon and confirm what you have:
\nexport XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user start docker.service
docker -c rootless info --format '{{.ServerVersion}} {{.SecurityOptions}}'
29.8.1 [name=seccomp,profile=builtin name=rootless name=cgroupns]
name=rootless is the daemon confirming what it is.
The installer tells you to add ~/bin to your PATH. For day-to-day use you do not have to:\nthe system docker client talks to the rootless daemon perfectly well through the context, and\nevery command in this post was run that way. You need the PATH only for the setup and uninstall\nscripts ā and without it they fail with dockerd-rootless.sh needs to be present under $PATH,\nwhich tells you nothing about what is actually wrong.
Ask a container who it is, then look at that same process from outside:
\ndocker -c rootless run --rm alpine:3.20 id
docker -c rootless run -d --name u0 alpine:3.20 sleep 60
ps -o user=,uid=,cmd= -C sleep | grep 'sleep 60'
docker -c rootless run -d --name u1000 --user 1000:1000 alpine:3.20 sleep 120
ps -o uid=,cmd= -C sleep | grep 'sleep 120'
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h8-uid-mapping-d53136aff71ace37ef98f708404f45d5.png\")
uid=0(root) gid=0(root) groups=0(root),...
ubuntu 1000 sleep 60
100999 sleep 120
Read those three lines together, because they are the entire mechanism:
\nubuntu.That last number is 100000 + 1000 - 1 ā the start of your reserved block, offset by the\ncontainer's own ID, because container root already took the first slot by mapping to you.
Which gives you a test rather than a promise. Work it out from your own /etc/subuid line and\nsee whether the number matches. If you instead see plain 1000, the translation is not\nhappening and something is wrong with the setup.
Clean up:
\ndocker -c rootless rm -f u0 u1000
Back to the command this post opened with. Same image, same mount of the entire filesystem,\nsame flags. Only the daemon differs:
\ndocker -c rootless run --rm -v /:/host alpine:3.20 \\
sh -c 'touch /host/root/rootless-demo || echo DENIED'
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h8-denied-e38f44dfdd6d58bf3f217438f3a584a0.png\")
touch: /host/root/rootless-demo: Permission denied
DENIED
The container still believes it is root. The kernel disagrees, because the process behind it\nbelongs to ubuntu, and ubuntu cannot write to /root.
That is the post, in two outputs.
\nFour things, all measured here rather than quoted.
\nContainers do not start noticeably slower. Three consecutive runs:
\nfor i in 1 2 3; do /usr/bin/time -f \"run+exit: %es\" \\
docker -c rootless run --rm alpine:3.20 true; done
docker -c rootless run --rm -p 80:80 alpine:3.20 true
![\"Three](\"https://development-wec.wiline.com/docs/assets/images/h8-costs-03291fa4bb34544ae9eff02c34a5fa7a.png\")
run+exit: 1.19s
run+exit: 0.96s
run+exit: 0.97s
Ports below 1024 are refused, and unusually, the error tells you every way out:
\ncannot expose privileged port 80, you can add
'net.ipv4.ip_unprivileged_port_start=80' to /etc/sysctl.conf (currently 1024),
or set CAP_NET_BIND_SERVICE on rootlesskit binary,
or choose a larger port number (>= 1024)
-p 18080:80 binds without complaint. Behind a reverse proxy this rarely matters.
Disk throughput limits stop working. The daemon says so at startup, and it is easy to miss:
\nWARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
So --device-read-bps and --device-write-bps silently do nothing. If you were relying on\nthose to stop one container starving the others, rootless takes that away.
Capability dropping still works, which matters because these are often treated as\nalternatives:
\ndocker -c rootless run --rm --cap-drop=ALL alpine:3.20 sh -c 'echo cap-drop ok'
cap-drop ok
They stack. Keep using user: and --cap-drop=ALL ā each layer assumes the one before it\nfailed.
This is the part that catches people moving a real service across. Host permissions apply as\nyou, not as root:
\nD=/tmp/rl-vol; mkdir -p $D; echo hello > $D/mine.txt
sudo sh -c \"echo secret > $D/theirs.txt; chmod 600 $D/theirs.txt\"
docker -c rootless run --rm -v $D:/data alpine:3.20 sh -c \\
'cat /data/mine.txt; cat /data/theirs.txt 2>&1 || echo DENIED'
docker -c rootless run --rm -v $D:/data alpine:3.20 stat -c '%u:%g %n' /data/mine.txt
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h8-volume-permissions-b16a91d894e4363082f108e17d3593e6.png\")
hello
cat: can't open '/data/theirs.txt': Permission denied
DENIED
0:0 /data/mine.txt
Your own file reads fine. The root-owned one is refused ā even though the container thinks it\nis root, because the kernel knows better. And your file appears inside the container as owned\nby 0:0, the same translation running in reverse.
Worth internalising before you move a database over and spend an hour wondering why it cannot\nopen its own data directory.
\nOne more: the rootless daemon keeps a separate image store at ~/.local/share/docker.\nImages your system daemon already has are invisible to it, and everything gets pulled again. On\na tight disk, budget for it.
For symmetry, the uninstall has the same guard:
\nPATH=/home/ubuntu/bin:$PATH ~/bin/dockerd-rootless-setuptool.sh uninstall
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h8-uninstall-refuses-9d0ed4f387ceb4f76df893f91156189a.png\")
[ERROR] Aborting because rootful Docker (/var/run/docker.sock) is running and accessible.
Set --force to ignore.
Add --force when you genuinely want it gone.
It changes the worst case from \"an attacker owns the machine\" to \"an attacker owns this\nuser account\". On a host where every service runs as ubuntu, that is a large move and not a\ncomplete one ā your SSH keys, your .env files and your source code all still sit inside that\naccount.
So the honest summary: rootless raises the floor. It does not make the room safe. Narrowing\nwhat the account itself can reach is the next rung, and that is where system-call filtering and\nimage supply chain come in.
\nBut the specific thing that was true at the top of this post ā that anyone able to run a\ncontainer could take the whole machine ā is no longer true, and you can prove it in one\ncommand.
\nsubuid(5) and user_namespaces(7)Agent orchestration part 4 split a toolbox so the\nscheduler could not issue refunds. Part 5\ngave the agent an identity at the gateway. Part 6 put a JWT\nverifier in front of the tools server so it refuses anonymous callers.
\nEvery one of those controls what the agent may call. Not one of them controls where the\nagent may go.
\nThat distinction is the whole of this post. Exfiltration does not need a tool. It needs a\nsocket.
\n\nOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Ubuntu 22.04, kernel 5.15. Docker\n29.1.3 with firewall backend iptables ā which on this distribution is iptables-nft\nv1.8.7, a shim writing into nftables. Sandbox container is alpine:3.20. UFW is active.
Every status code, error string and timing in this post is from the run described. Plain\nHTTP on a private LAN where the LAN appears, as throughout this series.
A firewall decides which packets may cross a boundary. Most of the time people mean it in one\ndirection ā ingress, who out there may reach in to your service. That is what publishing\na port does, and what part 1 of this series\nwas about.
\nEgress is the other direction: where your service may reach out to. Which addresses it\nmay open a connection to, and which it may not.
\nHardly anyone configures egress, and for an ordinary web application that is a reasonable\nchoice. It talks to its database and a payment provider, it does what its code says, and its\ncode does not change between deployments.
\nAn agent is not that. An agent decides what to do at runtime, from text it has just read ā an\nemail, a support ticket, a web page, a document somebody else wrote. If an attacker controls\nthat text, they influence what the agent does next. And the thing you do not want it doing next\nis exfiltration: sending your data somewhere it should not go.
\nExfiltration needs no special tool. Any process that can open a socket can send anything it can\nread. Which is why the direction nobody guards is the one that matters here.
\nThe leading reference for this class of problem is OWASP's LLM06:2025 Excessive Agency.\nIts prevention section lists, in order: minimize extensions, minimize extension\nfunctionality, avoid open-ended extensions, and further items about permissions and human\napproval.
\nRead the page looking for the network and it is not there. Not \"egress\", not \"firewall\", not\n\"network\", not \"isolation\", not \"sandbox\" ā the words do not appear in the entry at all.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/h7-owasp-mitigations-d1025852bd1958d91bfe4304dd74c563.png\")
Every mitigation on that list operates on tools. Remove the extension, narrow the\nextension, avoid the open-ended extension. All good advice. None of it changes a single byte\nof what a compromised process can do with a TCP socket.
\nThere is an argument that this is a gap rather than an oversight, and it is being made inside\nOWASP's own repository. An open issue proposing a runtime-enforcement mapping\nfor the Agentic Top 10 states it plainly:
\n\n\nThe enforcement pipeline assumes agents cannot reach tools without passing through the\nproxy. If tools are directly reachable, runtime enforcement collapses.
\n
and concludes that \"Network-level controls (e.g., Kubernetes NetworkPolicy, Docker network\nisolation) are required to close this gap in production.\"
\nNote what that is: an open issue, filed by a contributor, with its mapping marked\n(Proposed). It is not the standard. It is someone arguing the standard should say this.
\nAnd the reason this matters is not theoretical. EchoLeak (CVE-2025-32711)\nis a zero-click indirect prompt injection against Microsoft 365 Copilot, recorded in NVD as\n\"Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information\nover a network.\" Over a network. The tool list was never the thing that failed.
\niptables firewall backend ā the default. Check with\ndocker info | grep -i \"Firewall Backend\"sudo, and a willingness to write firewall rules on a box you can still reachDOCKER-USER is real firewall state on your host. A rule that matches more than you intended\nwill take services offline. Every rule in this post is scoped with -i br-agent, an interface\nthat belongs to one purpose-built network ā which is exactly why we name that bridge by hand\nin step 1 instead of letting Docker generate one.
If you are working over SSH, none of these rules touch the INPUT chain, so your session is\nnot at risk. Confirm that for yourself before you believe it.
A network, with the bridge interface named explicitly. Docker documents the option as\n\"Interface name to use when creating the Linux bridge\":
\ndocker network create --driver bridge \\
-o com.docker.network.bridge.name=br-agent agent-egress
ip -br link show br-agent
br-agent DOWN 8a:77:65:54:9b:0d <NO-CARRIER,BROADCAST,MULTICAST,UP>
DOWN, because nothing is plugged into it yet. A Docker network is an interface that does\nnot come up until a container attaches, and a rule matching -i br-agent on an idle network\nsilently matches nothing. Worth knowing before you spend an hour debugging a policy that was\nnever exercised.
The sandbox ā pinned, not latest:
docker run -d --name agent-sandbox --network agent-egress alpine:3.20 sleep infinity
docker exec agent-sandbox apk add --no-cache curl bind-tools
Now four destinations. Note what is not in this container: no agent, no framework, no MCP\nclient, no tools. Tool scoping is not being bypassed here ā there is nothing to bypass.
\ndocker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'inference API HTTP %{http_code}\\n' https://inference.wiline.com/v1/models
docker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'arbitrary host HTTP %{http_code}\\n' https://example.com
docker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'authentik LAN HTTP %{http_code}\\n' http://10.80.4.212:9100/
docker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'other network exit=%{exitcode} %{errormsg}\\n' http://172.21.0.2:5432/
inference API HTTP 401
arbitrary host HTTP 200
authentik LAN HTTP 302
other network exit=28 Connection timed out after 5002 milliseconds
![\"Four](\"https://development-wec.wiline.com/docs/assets/images/h7-before-unrestricted-527fc904745f99b3fdab1b9d08035428.png\")
The 401 is the useful one to understand. No API key was sent, so the request never reaches\nanything that could charge for it ā but a 401 is an HTTP response, which means the TCP\nhandshake completed, TLS negotiated, and a real server answered. Reachability, proven, for\nfree.
The fourth line is the one that keeps this honest. That is a Postgres container on a different\nDocker network, and it times out ā part 1 of this series\nestablished that separate bridge networks genuinely isolate, and this confirms it.
\nSo the shape of the hole is precise rather than vague:
\n| Destination | Result | |
|---|---|---|
| Another Docker network | timeout | Docker already handles this |
| Your own LAN, including the identity provider | 302 | open |
| The open internet | 200 | open |
| The one API it actually needs | 401 | open |
Docker isolates containers from each other and stops there. The two directions that matter\nfor exfiltration ā your network, and the internet ā are wide open, and no Docker guide frames\nthat as a gap because, for a web application, it isn't one.
\nOne rule. Note -A rather than -I: the deny goes at the bottom of the chain and stays\nthere, and every allow added later stacks above it. That ordering is the policy.
sudo iptables -A DOCKER-USER -i br-agent -j DROP
DOCKER-USER is where this belongs, and Docker's documentation is explicit about why:
\n\nRules appended to the
\nFORWARDchain will be processed after Docker's rules.
DOCKER-USER is processed before them ā the docs describe it as \"a placeholder for\nuser-defined rules that will be processed before rules in the DOCKER-FORWARD and DOCKER\nchains.\" Anywhere else and Docker's own ACCEPT gets there first.
Read that left to right and the placement explains itself. DOCKER-FORWARD holds one\nunconditional ACCEPT per bridge ā that is what makes container egress work at all, and it is\nwhy a rule appended to FORWARD never fires. DOCKER-USER is the only hook that sits upstream\nof it.
Re-run the same four probes:
\ninference API exit=28 Resolving timed out after 8000 milliseconds
arbitrary host exit=28 Resolving timed out after 8002 milliseconds
authentik LAN exit=28 Connection timed out after 8003 milliseconds
other network exit=28 Connection timed out after 8002 milliseconds
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h7-default-deny-dd96eebe9dcfc37644f95bf30a45d1ae.png\")
Read the error text, not the failure. Two different messages, and the difference is free\ndiagnostic information:
\nThe two by-name probes died at resolution; the two by-IP probes died at the socket. Which\ntells us something not obvious: Docker's embedded resolver forwards its upstream queries\nacross DOCKER-USER. The resolver lives at 127.0.0.11 inside the container's namespace,\nand it is tempting to assume that traffic never touches the host's forward path. It does.
Also note that every failure took the full eight seconds. A DROP produces timeouts; a\nREJECT fails instantly. Stealth against debuggability, and this post chooses stealth ā but\nif you are still iterating on a policy, REJECT will save you a great deal of waiting.
Before writing a rule for the resolver, find out which resolver the container actually uses:
\ndocker exec agent-sandbox cat /etc/resolv.conf
nameserver 127.0.0.11
search corp.internal internal mesh.example.com
options edns0 trust-ad ndots:0
# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [8.8.8.8 8.8.4.4]
ExtServers: [8.8.8.8 8.8.4.4]. The container is not using this network's DNS. The host\nresolves through systemd-resolved on 127.0.0.53, which is a loopback address and therefore\nuseless inside a container's namespace, so Docker substituted Google Public DNS without\nmentioning it.
Look at the inherited search domains in the same block (redacted here, but real on your box).\nEvery internal hostname that container looks up is being asked of 8.8.8.8, suffixed with the\nprivate domains it inherited from the host. Nobody chose that, and nothing surfaces it except\nreading this file.
It also decides the rule: allow the resolver the container has, not the one you assumed.
\ngetent hosts inference.wiline.com
67.207.107.229 inference.wiline.com
sudo iptables -I DOCKER-USER -i br-agent -p udp --dport 53 -d 8.8.8.8,8.8.4.4 -j ACCEPT
sudo iptables -I DOCKER-USER -i br-agent -p tcp --dport 53 -d 8.8.8.8,8.8.4.4 -j ACCEPT
sudo iptables -I DOCKER-USER -i br-agent -p tcp --dport 443 -d 67.207.107.229 -j ACCEPT
sudo iptables -S DOCKER-USER
-N DOCKER-USER
-A DOCKER-USER -d 67.207.107.229/32 -i br-agent -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER-USER -d 8.8.4.4/32 -i br-agent -p tcp -m tcp --dport 53 -j ACCEPT
-A DOCKER-USER -d 8.8.8.8/32 -i br-agent -p tcp -m tcp --dport 53 -j ACCEPT
-A DOCKER-USER -d 8.8.4.4/32 -i br-agent -p udp -m udp --dport 53 -j ACCEPT
-A DOCKER-USER -d 8.8.8.8/32 -i br-agent -p udp -m udp --dport 53 -j ACCEPT
-A DOCKER-USER -i br-agent -j DROP
Five allows on one deny. The same four probes, a third time:
\ninference API HTTP 401 exit=0
arbitrary host exit=28 Connection timed out after 8002 milliseconds
authentik LAN exit=28 Connection timed out after 8003 milliseconds
other network exit=28 Connection timed out after 8001 milliseconds
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h7-allowlist-applied-71e84e955ac296d1b262e120254f5409.png\")
Look at line two. It says \"Connection timed out\", not \"Resolving timed out\". DNS is\nopen, so example.com resolved to a real address ā and then the packet died. The lookup\nsucceeded and the socket did not.
That is the whole post in one line. A compromised process inside that container can discover\nwhere to send your data and cannot send it.
\nEvery guide on this subject tells you to accept RELATED,ESTABLISHED traffic first, and\nDocker's documentation shows the rule. This policy has no such rule, and the 401 above\nproves the full TLS handshake and HTTP response came back anyway.
The reason is the direction. Our drop matches -i br-agent ā traffic arriving from the\ncontainer. Return packets carry -o br-agent, never match it, pass through DOCKER-USER\nuntouched, and are accepted downstream by Docker's own connection-tracking rules.
You need the conntrack accept when your drop can match the return direction, which is the case\nin the ingress examples that advice comes from. A direction-specific egress drop does not. Add\nit anyway if you like ā it costs nothing ā but understand that copying it without understanding\nit is how people conclude firewall rules are black magic.
A policy nobody measures is a policy someone will disable under load. So: what does six rules\ncost?
\nThe obvious approach, twenty connects with the rules on, then twenty with them off:
\nfor i in $(seq 20); do docker exec agent-sandbox curl -sS -o /dev/null -w '%{time_connect}\\n' -m 8 https://inference.wiline.com/v1/models; done | sort -n | awk '{a[NR]=$1} END{printf \"rules ON ā median %.4fs min %.4fs max %.4fs (n=%d)\\n\", a[int(NR/2)+1], a[1], a[NR], NR}'
sudo iptables -F DOCKER-USER
rules ON ā median 0.1573s min 0.0940s max 0.3768s (n=20)
rules OFF ā median 0.0936s min 0.0911s max 0.1576s (n=20)
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h7-overhead-naive-f360e40c29ad6c2b0c7ff16a3663104c.png\")
64 milliseconds. Which would be a serious finding, if it were true.
\nIt is not. Six rules of packet matching is microseconds of kernel work, and the minimums give\nthe game away: 0.0940 against 0.0911, near-identical. The fast path is the same in both\nruns. The entire difference lives in the tail.
curl's time_connect is measured from the start of the request and therefore includes name\nresolution. The first run paid a full round trip to 8.8.8.8. By the second, the resolver\nhad the answer cached. We measured the order we ran them in.
Measure the TCP connect alone, by subtracting the lookup:
\nfor i in $(seq 30); do docker exec agent-sandbox curl -sS -o /dev/null -w '%{time_namelookup} %{time_connect}\\n' -m 8 https://inference.wiline.com/v1/models; done | awk '{print ($2-$1)}' | sort -n | awk '{a[NR]=$1} END{printf \"connect-only median %.5fs min %.5fs max %.5fs (n=%d)\\n\", a[int(NR/2)+1], a[1], a[NR], NR}'
rules ON ā connect-only median 0.07098s min 0.06999s max 0.08607s (n=30)
rules OFF ā connect-only median 0.07102s min 0.06996s max 0.07214s (n=30)
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h7-overhead-94f3b889647c2c273680e82c050b345a.png\")
A median difference of ā0.04 ms. Negative: the firewalled path measured marginally\nfaster, which is what noise looks like. Everything sits inside about Ā±40 Āµs.
\nSo the honest answer is not a number with a plus sign in front of it. It is that the cost of\nthis policy is below what this method can resolve ā and that the naive version of the same\nmeasurement was wrong by three orders of magnitude, in the direction that would have talked\nyou out of deploying it.
\niptables rules live in memory. That -F in the last step did to the policy exactly what a\nreboot does.
The reflex is iptables-persistent, and it is the wrong tool here: it saves the whole table\nincluding every rule Docker generated, and Docker regenerates those itself at start. You get\nduplicates and an ordering nobody chose.
What you want is a unit that owns only your rules and runs after Docker:
\n#!/usr/bin/env bash
# Egress allowlist for the agent bridge. Re-applied after Docker starts.
set -euo pipefail
BR=br-agent
DNS=\"8.8.8.8 8.8.4.4\"
API=$(getent hosts inference.wiline.com | awk '{print $1; exit}')
# Remove only our own rules, so this is safe to run twice.
# `|| true` matters: on a clean chain grep exits 1 and would abort the script.
existing=$(iptables -S DOCKER-USER | grep \" -i ${BR} \" || true)
if [ -n \"${existing}\" ]; then
printf '%s\\n' \"${existing}\" | sed 's/^-A /-D /' |
while read -r rule; do iptables ${rule} || true; done
fi
for d in ${DNS}; do
iptables -I DOCKER-USER -i \"${BR}\" -p udp --dport 53 -d \"${d}\" -j ACCEPT
iptables -I DOCKER-USER -i \"${BR}\" -p tcp --dport 53 -d \"${d}\" -j ACCEPT
done
iptables -I DOCKER-USER -i \"${BR}\" -p tcp --dport 443 -d \"${API}\" -j ACCEPT
iptables -A DOCKER-USER -i \"${BR}\" -j DROP
[Unit]
Description=Egress allowlist for the agent bridge
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/agent-egress.sh
[Install]
WantedBy=multi-user.target
sudo chmod +x /usr/local/sbin/agent-egress.sh
sudo systemctl daemon-reload && sudo systemctl enable --now agent-egress.service
Prove it is idempotent by restarting it and comparing the chain to itself:
\nsudo systemctl restart agent-egress.service && sudo iptables -S DOCKER-USER
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h7-persistent-unit-08c4df2a28634e493a6a37476e9429fb.png\")
Six rules, twice, no duplicates.
\nTwo things this buys quietly. getent runs on every start, so the API address is re-resolved\nat boot ā not continuous, but the cheap majority of the moving-target problem with no extra\ndaemon. And the delete loop strips only rules matching -i br-agent, so the unit never touches\nanything else in the chain.
You can put a default-deny egress policy in front of a container, allow exactly the destinations\nit needs, tell a DNS failure apart from a socket failure by reading the error, measure the cost\nwithout fooling yourself, and make the whole thing survive a reboot.
Docker 29 ships an nftables firewall backend alongside iptables. It is not the default and this\npost did not run it ā the switch rebuilds every rule on the host, and verifying the full\nbehaviour needs a reboot. What follows is from Docker's documentation, flagged as such.
\nEverything above stops applying, because:
\n\n\nIn Docker's nftables implementation, there is no
\nDOCKER-USERchain.
You create your own table instead, and the priority does the work DOCKER-USER used to:
\n\nIf your rules need to run before Docker's rules, give the base chains a lower priority number\nthan Docker's chain.
\n
And a rule that would be final under iptables is not:
\n\n\nIn nftables, an \"accept\" rule is not final. It terminates processing for its base chain, but\nthe accepted packet will still be processed by other base chains, which may drop it.
\n
Overriding Docker's drop then requires --bridge-accept-fwmark.
The trap worth knowing about is the transition, and it is quiet in both directions:
\n\n\nWhen starting the daemon with nftables after running with iptables, Docker will not remove the\njump from the
\nFORWARDchain toDOCKER-USER. So, rules created inDOCKER-USERwill\ncontinue to run until the jump is removed or the host is rebooted. When starting with nftables,\nthe daemon will not add the jump. So, unless there is an existing jump, rules inDOCKER-USER\nwill be ignored.
Switch backends and your policy keeps working ā until the next reboot, when it silently stops.\niptables -S DOCKER-USER will still list every rule.
status=1/FAILURE with nothing else in the journal. The cleanup line is the cause: on an empty\nchain, grep finds no matching rules and exits 1, which set -euo pipefail turns into an\nabort before any rule is added. The idempotency guard breaks the one run that needs no\nidempotency. See the || true in step 5.
Check whether the failure says \"Resolving\" or \"Connection\". If it says Resolving, your DNS\nrule is the problem and the destination rule was never reached. Containers frequently do not\nuse the resolver you expect ā read /etc/resolv.conf inside the container, not on the host.
Check that the bridge is up: ip -br link show br-agent. A network with no attached container\nhas a DOWN interface, and a rule matching -i on it matches nothing.
That is DROP behaving correctly. Use REJECT while iterating if you would rather fail fast.
You are measuring DNS. Use %{time_connect} minus %{time_namelookup}, and be suspicious of\nany result where the minimums match but the medians don't.
Done: a container that can reach exactly one destination and a resolver; an exfiltration attempt\nthat fails after a successful DNS lookup rather than before it; a policy that reapplies itself\nafter a reboot and re-resolves its target when it does; and a measured cost indistinguishable\nfrom zero.
\nNot done:
\n8.8.8.8, and data can be\nsmuggled inside them. Closing that means a resolver you control and a rule pointing only at\nit ā which is a different post.ACCEPT in DOCKER-FORWARD.The obvious sequel is the resolver: run one you control, point the container at it, allow port\n53 only to that address, and the DNS channel closes along with the dependency on Google. It also\nsets up domain-based allowlisting properly, which is the piece of the moving-target problem this\npost left open.
\nPart 5 gave an agent its own identity at\nthe gateway: a token issued by authentik over client credentials, carrying a scope, expiring in\nfive minutes. It ended by naming what it had not covered ā the MCP tools server from\nagent orchestration part 4 still trusts anything that\ncan reach its port, issue_refund included.
That post was explicit about the limit of what it had built:
\n\n\nThe MCP server still trusts everyone. Scoping happens in the client. Anything that can\nreach
\n127.0.0.1:8770can callissue_refunddirectly, agent or not.
Splitting the toolbox per role stopped an agent from reaching a tool it shouldn't. It did\nnothing about a curl. This part closes that.
The same box as Parts 3 through 5: one WEC Instance, 8 vCPU AMD EPYC 7601, 15 GB RAM, Ubuntu\n22.04, Python 3.10. authentik 2026.8.1 on host port 9100. fastmcp 4.0.2, mcp 2.1.1,\nlangchain 1.4.0. The MCP server is office_tools.py from\nagent orchestration part 3, unchanged except\nfor the lines shown here, still on 127.0.0.1:8770.
Plain HTTP on a private LAN, as throughout this series.
The obvious move is to reuse Part 5's agent-gateway client and add a second scope to it. It\nalready exists, the agent already holds its credentials, and it would work.
Don't. If one credential opens both the gateway and the tools server, then a leaked gateway key\nalso issues refunds, and the blast radius of losing it doubles. The whole thread running through\nPart 4's groups and Part 5's grant types is that a credential should do one job.
\nSo the tools server gets its own client, its own scope, and ā because in authentik every\nprovider is its own issuer ā its own issuer. That last part turns out to matter more than\nexpected, and we get to it in Step 4.
\nagent-gateway\nclient from Part 5fastmcp 4.0.0 or newer ā the client-credentials helper discussed in Step 5 does not exist\nbefore itThis is Part 5's procedure with different names, so it is deliberately terse here. If any of it\nis unfamiliar, Part 5 walks through the same three screens\nwith figures.
\nThe provider. Applications ā Providers ā New Provider ā OAuth2/OpenID ā Next. Name\nagent-tools, Client Type Confidential, Redirect URIs empty, and in Grant Types uncheck\neverything except Client credentials.
The scope. Customization ā Property Mappings ā New Property Mapping ā Scope Mapping. Mapping\nName mcp-invoke, Scope name mcp:invoke, expression return {}. The two name fields are\ndifferent things: the first is authentik's label, the second is the string that lands in the\ntoken.
The application. Applications ā Applications ā New Application ā¾ ā with Existing\nProviderā¦, named Agent Tools, slug agent-tools typed by hand, provider agent-tools,\nhidden from the dashboard. Then Edit the provider ā Advanced protocol settings ā Scopes, and\nmove mcp-invoke into Selected.
Verify the grant types from the database rather than the form, which is\nPart 4's lesson:
\ncd ~/authentik && docker compose exec server ak shell -c \"
from authentik.providers.oauth2.models import OAuth2Provider
for p in OAuth2Provider.objects.all():
print(f'{p.name}: {p.grant_types}')\" 2>/dev/null | tail -5
Provider for Langfuse: ['authorization_code', 'implicit', 'urn:ietf:params:oauth:grant-type:device_code']
Provider for LiteLLM: ['authorization_code', 'implicit', 'hybrid', 'refresh_token', 'client_credentials', 'password', 'urn:ietf:params:oauth:grant-type:device_code']
agent-gateway: ['client_credentials']
agent-tools: ['client_credentials']
Two clients trimmed to one grant each, and ā visible in the same output ā the LiteLLM provider\nfrom Part 4 still carrying password and five others nobody chose. Part 5 said to go back and\nfix those. This is what not doing it looks like.
Read the credentials out and keep them beside the gateway's, in their own directory:
\ncd ~/authentik
read -r TID TSEC < <(docker compose exec -T server ak shell -c \"
from authentik.providers.oauth2.models import OAuth2Provider as P
p = P.objects.get(name='agent-tools')
print(p.client_id, p.client_secret)\" 2>/dev/null | tail -1)
mkdir -p ~/mcp-auth && cd ~/mcp-auth
umask 077
cat > .env <<EOF
AK_TOKEN_URL=http://10.80.4.212:9100/application/o/token/
AK_CLIENT_ID=$TID
AK_CLIENT_SECRET=$TSEC
EOF
cp ~/agent-auth/token.sh ~/mcp-auth/token.sh
The helper is Part 5's, unchanged ā it takes a scope and returns an access token.
\n~/mcp-auth/token.sh mcp:invoke | jq -R 'split(\".\")[1] | gsub(\"-\";\"+\") | gsub(\"_\";\"/\") | @base64d | fromjson'
{
\"iss\": \"http://10.80.4.212:9100/application/o/agent-tools/\",
\"sub\": \"545a532b77a0f877339a5e4312970069deeb9ded604494c510debdfdce9c970d\",
\"aud\": \"xNxmOEyPMPVThmxgLsyvTCHa3qvnpVjQSOY0bZck\",
\"exp\": 1789050030,
\"iat\": 1789049730,
\"scope\": \"mcp:invoke\"
}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h6-tools-token-611a3b28adc48e09f38c168c63c7d1ac.png\")
Three fields differ from the gateway's token and all three are load-bearing. The issuer ends\n/agent-tools/ rather than /agent-gateway/. The audience is a different client ID. The\nscope is mcp:invoke. exp minus iat is 300 seconds, same as Part 5 ā remember that\nnumber, it comes back in Step 5.
If the token comes back with \"scope\": \"\", the mapping exists but is not attached to the\nprovider. authentik's own help text on that pane: \"Select which scopes can be used by the\nclient. The client still has to specify the scope to access the data.\" Attaching makes it\navailable; the client still has to ask.
FastMCP validates JWTs with JWTVerifier, which takes the four things authentik just told us.\nBack up the server first, since it is the artefact two published parts are built on:
cp ~/mcp-tools/office_tools.py ~/mcp-tools/office_tools.py.bak
Its configuration goes in a file rather than the source:
\ncd ~/mcp-tools
umask 077
cat > .env.mcp <<'EOF'
MCP_JWKS_URI=http://10.80.4.212:9100/application/o/agent-tools/jwks/
MCP_ISSUER=http://10.80.4.212:9100/application/o/agent-tools/
MCP_AUDIENCE=<the agent-tools client id>
EOF
Then two changes to office_tools.py ā an import, and a verifier where the bare constructor was:
from fastmcp import FastMCP, Context
from fastmcp.server.auth.providers.jwt import JWTVerifier
verifier = JWTVerifier(
jwks_uri=os.environ[\"MCP_JWKS_URI\"],
issuer=os.environ[\"MCP_ISSUER\"],
audience=os.environ[\"MCP_AUDIENCE\"],
required_scopes=[\"mcp:invoke\"],
)
mcp = FastMCP(\"office-tools\", auth=verifier)
That is the entire server-side change. JWTVerifier fetches the JWKS, matches each token's\nkid to a key, and checks signature, issuer, audience, expiry and scope before any tool runs.
ssrf_safe and private addressesJWTVerifier takes an ssrf_safe argument that defaults to False, and enabling it sounds\nunambiguously like a good idea. On a setup like this one it breaks the verifier. FastMCP's SSRF\nguard rejects on IP, and its own error text is \"Private, loopback, link-local, and reserved IPs\nare not allowed\" ā the check explicitly covers 10.x, 172.16-31.x, 192.168.x and 127.x.\nOur JWKS is at 10.80.4.212, so the fetch never leaves the process.
If you want the guard on and your identity provider is genuinely internal, the module reads\nFASTMCP_SSRF_TRUST_PROXY, which skips DNS resolution and the IP blocklist entirely.
Restart with the new environment loaded:
\npkill -f office_tools.py
cd ~/mcp-tools && set -a && source .env.mcp && set +a && \\
nohup ./.venv/bin/python -u office_tools.py > server.log 2>&1 &
sleep 4 && tail -8 server.log
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h6-server-auth-startup-e2aacc2f7033430c7a21caae3e07b195.png\")
Read that startup log carefully, because of what is missing from it. It reports the transport,\nthe mode and the port, and says nothing whatsoever about authentication. A server with a\nverifier attached and a server without one log the same lines. There is no \"auth enabled\"\nmessage to look for, so the only way to know the verifier took is to make a request ā which is\nthe next step.
\nThe same shape as Part 5's three status codes, because it reads well and because the middle one\nis the interesting one:
\nTOOLS=$(~/mcp-auth/token.sh mcp:invoke)
GW=$(~/agent-auth/token.sh gateway:invoke)
req() {
curl -sS -o /dev/null -w \"$1 HTTP %{http_code}\\n\" -X POST http://127.0.0.1:8770/mcp \\
-H 'Content-Type: application/json' \\
-H 'Accept: application/json, text/event-stream' \\
${2:+-H \"Authorization: Bearer $2\"} \\
-d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}'
}
req \"no token \"
req \"gateway token\" \"$GW\"
req \"tools token \" \"$TOOLS\"
no token HTTP 401
gateway token HTTP 401
tools token HTTP 200
![\"Three](\"https://development-wec.wiline.com/docs/assets/images/h6-three-way-5268dcb96fa1731e359bf14df5c1b0e8.png\")
The first line is the gap agent orchestration part 4 admitted to, now closed. Before this change\nthat request returned the full five-tool catalogue to anybody who asked ā and with the right\ntoken, it still does:
\ncurl -sS -X POST http://127.0.0.1:8770/mcp \\
-H \"Authorization: Bearer $TOOLS\" \\
-H 'Content-Type: application/json' \\
-H 'Accept: application/json, text/event-stream' \\
-d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}' \\
| tr -d '\\r' | sed -n 's/^data: //p' | jq '.result.tools | length, (.[].name)'
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h6-tools-list-authorised-e04041592339f5df452a50d4fc2cd30d.png\")
Same five tools as before. Authentication changed who may ask, not what the server does.
\nThe second is the one worth pausing on. That is a valid, unexpired, correctly signed authentik\ntoken that works perfectly well against the gateway, and the tools server refuses it. Ask the\nserver why ā sending the request again first, so the answer is at the end of the log rather\nthan buried under whatever has hit the server since:
\nGW=$(~/agent-auth/token.sh gateway:invoke)
curl -sS -o /dev/null -w 'gateway token: HTTP %{http_code}\\n' -X POST http://127.0.0.1:8770/mcp \\
-H \"Authorization: Bearer $GW\" \\
-H 'Content-Type: application/json' \\
-H 'Accept: application/json, text/event-stream' \\
-d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}'
tail -12 ~/mcp-tools/server.log
WARNING Bearer token rejected for client aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ
: issuer mismatch (got 'http://10.80.4.212:9100/application/o/agent-gateway/',
expected 'http://10.80.4.212:9100/application/o/agent-tools/')
INFO Auth error returned: invalid_token (status=401)
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h6-issuer-mismatch-log-67e89b1720edb880e97f3f7258d47aac.png\")
It failed on iss, not on scope. That is the separate provider paying off. In authentik each\nprovider is its own issuer, so a token from the wrong client is rejected at the first gate,\nbefore audience or scope are looked at. Had we taken the easy road and added a second scope to\nthe gateway's client, the same request would have got several checks further before\nrequired_scopes turned it away. Same outcome, more surface.
Two other things in that log. It names the offending client by ID, and prints both the issuer it\ngot and the one it wanted ā so debugging is a tail, not a guess. And the caller receives only\ninvalid_token, which is correct: telling a client which issuer you expected is telling an\nattacker what to forge.
grep -iE 'invalid|token|401' server.log returns the last line and hides the first. You get\ninvalid_token (status=401) and conclude the server won't say why. It says why, on the line\nabove, wrapped across four lines that your pattern doesn't match. Read it unfiltered.
The tools server is now protected, which means every script from\npart 3 and\npart 4 is broken. They connect with no token.
\nFastMCP documents exactly the right thing for this: ClientCredentialsOAuthProvider, which runs\nthe client-credentials grant itself and ā the part that matters given our 300-second tokens ā\n\"when the token expires it is re-acquired automatically on the next request.\"
It does not work here. Try it and you get:
\nmcp.client.auth.exceptions.OAuthTokenError: Token exchange failed (404): Not Found
The server log explains what actually happened:
\nGET /.well-known/oauth-authorization-server 404
POST /token 404
GET /.well-known/oauth-protected-resource 404
GET /.well-known/oauth-protected-resource/mcp 404
GET /.well-known/openid-configuration 404
The provider takes the MCP server's URL, not a token endpoint, because \"the token endpoint\nis discovered from the server's OAuth metadata.\" Our server publishes none ā and FastMCP's own\ndocumentation says so, on a different page: \"TokenVerifier focuses exclusively on token\nvalidation without providing OAuth discovery metadata.\"
So the two pages are consistent and the combination is a documented dead end. You just have to\nread both to find that out, and the failure tells you none of it: four 404s you only see if you\nare watching the server, a fallback that POSTs /token at the resource server, and an error\nthat never mentions discovery.
The path the docs point to instead is the client obtaining its token separately. BearerAuth\ntakes a token you already have and attaches it to every request ā worth proving standalone\nbefore wiring it into four scripts:
![\"A](\"https://development-wec.wiline.com/docs/assets/images/h6-client-bearer-works-5250c1328730dcadf4366a6843f881c0.png\")
That works, so it can be shared. One helper, used by every script in the series:
\n\"\"\"One authenticated MCP client, shared by every agent in the series.\"\"\"
import os
import subprocess
from fastmcp import Client
from fastmcp.client.auth import BearerAuth
MCP_URL = \"http://127.0.0.1:8770/mcp\"
TOKEN_SH = os.path.expanduser(\"~/mcp-auth/token.sh\")
def fetch_token(scope: str = \"mcp:invoke\") -> str:
\"\"\"Mint a short-lived token from authentik via the client credentials grant.\"\"\"
out = subprocess.run([TOKEN_SH, scope], capture_output=True, text=True, check=True)
return out.stdout.strip()
def authed_client() -> Client:
return Client(MCP_URL, auth=BearerAuth(token=fetch_token()))
Each agent then changes by two lines ā an import, and what it hands MCPAdapter:
from mcp_auth import authed_client
async with MCPAdapter(authed_client()) as adapter:
MCPAdapter accepts a configured Client as readily as a URL string, which\npart 3 used for caching without needing it\nfor anything else. Here it is what lets the token in.
BearerAuth is the explicit form. FastMCP also accepts the token as a bare string ā\nClient(url, auth=token) ā and adds the scheme itself; its documentation is specific that you\n\"do not include the Bearer prefix\" if you do. The class is worth the extra import here only\nbecause it makes the intent obvious at the call site.
BearerAuth holds one fixed string. A token minted when the agent starts is dead 300 seconds\nlater, and nothing re-acquires it ā the automatic renewal was the one thing\nClientCredentialsOAuthProvider would have given us. For a short run it does not matter. For an\nagent that waits on a human, as\npart 3's refund does, it certainly can. Either\nraise the token lifetime on the provider, or mint per call rather than per client.
Discovery is not execution. A token that lists tools might still fail at the call, so check the\nthing that touches the database:
\nTOOLS=$(~/mcp-auth/token.sh mcp:invoke)
curl -sS -X POST http://127.0.0.1:8770/mcp \\
-H \"Authorization: Bearer $TOOLS\" \\
-H 'Content-Type: application/json' \\
-H 'Accept: application/json, text/event-stream' \\
-d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"find_customer\",\"arguments\":{\"query\":\"Maria\"}}}' \\
| tr -d '\\r' | sed -n 's/^data: //p' | jq -c '.result.content[0].text // .error'
\"[{\\\"id\\\":1,\\\"name\\\":\\\"Maria Alvarez\\\",\\\"email\\\":\\\"maria@example.com\\\"}]\"
![\"A](\"https://development-wec.wiline.com/docs/assets/images/h6-tool-call-authorised-d181de9df9cf4ce219d07511a7b381d2.png\")
Drop the Authorization header and the same call returns 401. A real row with a token, nothing\nwithout ā that is authentication reaching execution, not just the catalogue.
You can put a JWT verifier in front of an MCP tools server so it refuses anonymous callers and\nrefuses valid tokens issued to a different client ā and read the server log to know which of\nissuer, audience, scope or expiry did the refusing.
Token exchange failed (404): Not FoundāClientCredentialsOAuthProvider against a JWTVerifier server. Discovery found nothing and it\nfell back to guessing. See Step 5; use BearerAuth.
2 validation errors ... Missing required argumentāThe token was fine ā this comes from inside the tool. params.name is the tool's name,\nparams.arguments are its parameters, and both nest a field called name. find_customer\ntakes query.
invalid_tokenāYou grepped it. See the warning in Step 4.
\nExpected. They connect without a token. Step 5.
\nDone: an MCP server that authenticates every caller, a credential that is not the gateway's, a\nrefusal that happens at the issuer rather than deep in the scope check, and a log that names\nwhich client was turned away and why.
\nNot done:
\nissue_refund to anything that isn't billing.\" Half of\nthat is now true: the call is authenticated. The other half is not. required_scopes is a\nproperty of the server, not of a tool, so every holder of mcp:invoke can call issue_refund\nexactly as easily as find_customer, and the server still cannot tell the scheduler from the\nbilling agent. Splitting tools per agent, as\npart 4 did, remains a client-side arrangement.Per-tool authorisation is the obvious gap and it does not have an obvious answer. required_scopes\ngates the server; gating issue_refund differently from find_customer means either a second\nserver with its own client, or authorisation inside each tool body reading the verified claims.\nThe first is more boxes and a clean boundary. The second is a check on every tool you ever write,\nwhich is precisely the failure mode\npart 4 warned about for policy-in-the-tool.
![\"LiteLLM\"](\"https://development-wec.wiline.com/docs/assets/images/litellm-wordmark-86d6f0432720b27534e05bc579dfc29c.png\")
Part 4 put group access control on the\ngateway's admin UI and then, at the end, called the API from a shell with no account, no\nsession and no group. It answered normally. The conclusion was that SSO protects a control\nplane and the data plane authenticates machine callers with keys ā which it has to, because\nan agent running at 3am cannot complete a browser login.
\nThat was true and it was also a stopping point rather than an answer. \"Machine callers use\nkeys\" leaves you with a credential that never expires, that no identity provider knows about,\nand that survives the person who created it. Part 4 said so plainly: removing someone from a\ngroup does not revoke their keys, a leaked key is unaffected by identity entirely, and keys\noutlive people.
\nThis part gives the machine an identity instead of a key.
\nauthentik issues the agent a token over the client credentials grant ā no browser, no\nconsent screen, no human. The token is signed, carries a scope, and expires in five minutes.\nThe gateway verifies it locally against authentik's public keys and refuses anything without\nthe right scope. At the end, three status codes show the boundary holding.
\n\nOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Docker 29.1.3, kernel 5.15. authentik\n2026.8.1, the same instance from Parts 3 and 4 on host port 9100. LiteLLM 1.96.2 from\nPart 1 of the gateway series, bound to\n127.0.0.1:4000, models served over WEC Inference. Verification uses PyJWT 2.13.0 and\ncryptography 50.0.0, both already present in the LiteLLM image.
Plain HTTP on a private LAN, as in Parts 3 and 4. Substitute your own addresses.
Parts 3 and 4 used the authorization code grant throughout. A person clicks a button, gets\nredirected to authentik, types a password, sees a consent screen, and is redirected back with\na code the application exchanges for a token. Every step of that assumes a browser and a human\nattached to it.
\nThe client credentials grant answers a different question. There is no user. The client\nis the principal, and it proves that with an ID and a secret, in one request, with no\nredirect anywhere.
\n\nThe property worth noticing: the gateway never asks authentik whether a token is good. It\nfetches authentik's public keys once, caches them, and checks the signature itself. Identity\nscales without the identity provider becoming a bottleneck ā or a single point of failure on\nevery request.
\nLITELLM_MASTER_KEY, and shell access to the boxApplications ā Providers ā New Provider ā OAuth2/OpenID Provider ā Next.
\nagent-gatewaydefault-provider-authorization-explicit-consent. It is\nrequired, and it is inert here: it governs the consent screen a human sees, and no human\nwill ever use this client. You have to pick one anyway.Then scroll to Grant Types, and look at what a new provider ships with.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-grant-types-default-a874ef714983fd8cd1c64bf2b966cb09.png\")
Seven of eight, enabled by default. Two of them ā Implicit and Password ā did not make\nit into OAuth 2.1, which states plainly that \"some features available in OAuth 2.0, such as the\nImplicit or Resource Owner Credentials grant types, are not specified in OAuth 2.1\"\n(Ā§10.1). Password\nin particular has the client handle a user's actual password, which is why it fell out of\nfavour. Nothing asked whether you wanted them and nothing warns you that they are on.
\nUncheck everything except Client credentials.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-grant-types-scoped-d5992f14733c7ab87134b9a587a05c87.png\")
This is the same least-privilege reflex Part 4 applied to group membership, one layer down. A\nclient that only ever uses one grant should only be permitted one grant, so that a\nmisconfiguration elsewhere cannot turn it into a password-accepting endpoint.
\nThose defaults applied to every provider you created in Parts 3 and 4 too, and nothing has\ntrimmed them since. Ask the database rather than the UI:
cd ~/authentik && docker compose exec server ak shell -c \"
from authentik.providers.oauth2.models import OAuth2Provider
for p in OAuth2Provider.objects.all():
print(f'{p.name}: {p.grant_types}')\" 2>/dev/null | tail -5
Provider for Langfuse: ['authorization_code', 'implicit', 'urn:ietf:params:oauth:grant-type:device_code']
Provider for LiteLLM: ['authorization_code', 'implicit', 'hybrid', 'refresh_token', 'client_credentials', 'password', 'urn:ietf:params:oauth:grant-type:device_code']
agent-gateway: ['client_credentials']
The gateway's provider is still holding password and client_credentials\nalongside the grant it actually uses. Neither was needed, and the password grant in particular\nis a live way in that nobody put there on purpose. Trim each provider to the grants it uses.
Read authentik's own help text on the Redirect URIs field: \"If no explicit authorization\nredirect URIs are specified, the first successfully used authorization redirect URI will be\nsaved.\"
Empty does not mean \"deny all\". It means trust-on-first-use ā authentik pins whichever\nredirect URI shows up first and keeps it. That is harmless for a client which never redirects,\nwhich is our case. It is not harmless on an interactive provider, where it means you have no\nredirect validation at all until someone logs in and pins one for you. Part 3 warned against\nsetting the mode to .*; this is the quieter version of the same hazard.
Read back the credentials and try to get a token. It will not work yet, and the reason is the\nmost useful thing in this post.
\nmkdir -p ~/agent-auth
cd ~/agent-auth
umask 077
cat > .env <<EOF
AK_TOKEN_URL=http://10.80.4.212:9100/application/o/token/
AK_CLIENT_ID=<client id from the provider page>
AK_CLIENT_SECRET=<client secret from the provider page>
EOF
umask 077 creates the file with 600 permissions rather than chmod-ing it afterwards, so there\nis no window where the secret is world-readable.
A small helper, because you will fetch tokens constantly:
\ncat > ~/agent-auth/token.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
source \"$(dirname \"$0\")/.env\"
curl -sS -X POST \"$AK_TOKEN_URL\" \\
-d grant_type=client_credentials \\
-d client_id=\"$AK_CLIENT_ID\" \\
-d client_secret=\"$AK_CLIENT_SECRET\" \\
-d scope=\"${1:-}\" \\
| jq -r .access_token
EOF
chmod +x ~/agent-auth/token.sh
Ask for a token:
\ncd ~/agent-auth
source .env
curl -sS -X POST \"$AK_TOKEN_URL\" \\
-d grant_type=client_credentials \\
-d client_id=\"$AK_CLIENT_ID\" \\
-d client_secret=\"$AK_CLIENT_SECRET\" | jq
{
\"error\": \"invalid_grant\",
\"error_description\": \"The provided authorization grant or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client\",
\"request_id\": \"757c397b97a745259e288b6d4ff1553c\"
}
Every clause in that description is wrong for our situation. Nothing is expired, nothing is\nrevoked, no redirection URI was used, and the client is the one the credentials belong to. The\nserver log records the 400 and no reason. authentik's Events show nothing useful either.
\nThe actual cause is visible in the Providers list, which flags it plainly:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-provider-unassigned-f6f2e4435e386fcaa6db74ac117566a4.png\")
A provider with no application cannot issue a token. authentik authorizes against the\napplication, so a provider on its own has nothing to authorize against ā and says\ninvalid_grant instead of saying that.
This is not in authentik's client credentials documentation, which documents five ways to\nauthenticate ā three static-credential variants and two JWT ones ā and says nothing about\nneeding an application.
\nPair one. Note the route: the New Application wizard always creates a new provider, so it\ncannot adopt the one we just built.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-application-created-4cf5cd7238257dde87681d0c92e34efd.png\")
Its step list gives it away ā Choose a Provider is step 2, and every option there builds a\nnew one. The caret beside the button is where the other path lives.
\nApplications ā Applications ā New Application ā¾ ā with Existing Providerā¦
\nAgent Gatewayagent-gateway, typed explicitly. Part 4's finding was that authentik derives\nslugs from names and turned LiteLLM into lite-llm; the slug lands in the issuer URL, so\nit is worth setting by hand every time.agent-gatewayRetry the token request and it succeeds. Nothing changed except the pairing.
\nThe token you now get carries \"scope\": \"\". It proves who the caller is and grants no\nauthority at all. A gateway receiving it knows the client is agent-gateway and nothing more.
Customization ā Property Mappings ā New Property Mapping ā Scope Mapping.
\ngateway-invoke, authentik's label for the objectgateway:invoke, the string clients request and that appears in the tokenreturn {}![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-scope-mapping-93c6d51e31e1b4e0e089d72f023f4714.png\")
The two name fields are different things and the form does not make that obvious. The\nexpression is Python whose return value is merged into the token's claims; an empty dict is\nright here, because we want the scope named in the token, not extra data carried inside it.
\nAttach it: Applications ā Providers ā agent-gateway ā Edit ā Advanced protocol settings ā\nScopes, and move gateway-invoke from Available to Selected.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-provider-scopes-1cf3161c12ea30bfd90af1bc227bb8d1.png\")
Read the help text under that pane, because it is the whole mechanic:
\n\n\n\"Select which scopes can be used by the client. The client still has to specify the scope to\naccess the data.\"
\n
Attaching a scope makes it available. The client still has to ask. That is why the first\ntoken came back empty even though three scopes were already attached ā nothing had requested\nthem.
\nThe dual-list marks entries for deletion when you click them in the right-hand column, and the\nonly indication is a line of small text above it reading \"4 items selected. 1 item marked to\nremove.\" Saving at that moment silently unlinks the scope you just added. Read that line\nbefore you save.
Now ask for it:
\nTOKEN=$(~/agent-auth/token.sh gateway:invoke)
jq -R 'split(\".\")[1] | gsub(\"-\";\"+\") | gsub(\"_\";\"/\") | @base64d | fromjson' <<< \"$TOKEN\"
{
\"iss\": \"http://10.80.4.212:9100/application/o/agent-gateway/\",
\"sub\": \"2451c6a1bab3085590263f119e650448beb013336146bcfee8241dd5832c200b\",
\"aud\": \"aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ\",
\"exp\": 1788901322,
\"iat\": 1788901022,
\"acr\": \"goauthentik.io/providers/oauth2/default\",
\"jti\": \"NRDTukijwLJbW7gpPtn9zUZwzTj6d2kBSZCuFYvX\",
\"azp\": \"aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ\",
\"uid\": \"neugHWoWQOQzADgbV1vKwzew2COQ1RgCXfyywgqa\",
\"scope\": \"gateway:invoke\"
}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-token-scoped-83408e96e6123963ab117fedea8d70de.png\")
exp minus iat is 300 seconds. That is the entire argument for this over a static key: a\ntoken pasted into a chat window, a log file or a screenshot is worthless five minutes later.
There is no email or preferred_username in there, because there is no user. sub is a\nservice account authentik created for you, named ak-agent-gateway-client_credentials ā it\nappears in Directory ā Users without ever being asked for.
LiteLLM documents JWT authentication with a litellm_jwtauth block, enforce_scope_based_access\nand scope_mappings. Configure it on a stock install and nothing happens, because the same page\nsays:
\n\n\"JWT-based Auth requires a LiteLLM Enterprise license.\"
\n
Check before you write config:
\ndocker exec llm-gateway python -c \"
from litellm.proxy.proxy_server import premium_user
print('premium_user =', premium_user)\"
premium_user = False
False means that whole documented path is closed to you. The open-source alternative is\ncustom_auth, which hands every request to a function you write ā and for this job it is about\ntwenty-five lines.
Once custom_auth is set, every request goes through your function, including the sk-\nvirtual keys from the rest of the gateway series and the master key itself. A handler that only\nunderstands JWTs locks you out of your own gateway. The handler below falls back to the master\nkey deliberately.
Back up before you start: cp config.yaml config.yaml.bak
import os
import jwt
from fastapi import Request
from litellm.proxy._types import UserAPIKeyAuth
JWKS_URL = os.environ[\"JWT_PUBLIC_KEY_URL\"]
ISSUER = os.environ[\"JWT_ISSUER\"]
AUDIENCE = os.environ[\"JWT_AUDIENCE\"]
REQUIRED_SCOPE = \"gateway:invoke\"
MASTER_KEY = os.environ[\"LITELLM_MASTER_KEY\"]
_jwks = jwt.PyJWKClient(JWKS_URL)
async def user_api_key_auth(request: Request, api_key: str) -> UserAPIKeyAuth:
token = api_key.removeprefix(\"Bearer \").strip()
# Not a JWT: fall back to the master key so existing tooling keeps working.
if token.count(\".\") != 2:
if token == MASTER_KEY:
return UserAPIKeyAuth(api_key=token)
raise Exception(\"not a JWT and not the master key\")
signing_key = _jwks.get_signing_key_from_jwt(token)
claims = jwt.decode(
token,
signing_key.key,
algorithms=[\"RS256\"],
issuer=ISSUER,
audience=AUDIENCE,
)
if REQUIRED_SCOPE not in claims.get(\"scope\", \"\").split():
raise Exception(f\"token is missing scope {REQUIRED_SCOPE}\")
return UserAPIKeyAuth(api_key=token, user_id=claims[\"sub\"])
PyJWKClient does the part that looks hard. It fetches the JWKS, matches the token's kid\nheader to the right key, and caches it ā so after the first request, verification is local\narithmetic. jwt.decode checks the signature, the issuer, the audience and the expiry in one\ncall, and raises if any of them is wrong.
Three environment variables, appended to ~/llm-gateway/.env:
JWT_PUBLIC_KEY_URL=http://10.80.4.212:9100/application/o/agent-gateway/jwks/
JWT_ISSUER=http://10.80.4.212:9100/application/o/agent-gateway/
JWT_AUDIENCE=aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ
JWT_ISSUER and JWT_AUDIENCE must match the iss and aud you saw in the decoded token\nexactly, or every token is refused.
Mount the module the same way the gateway already mounts scrubber.py, in\ndocker-compose.yml:
- ./jwt_auth.py:/app/jwt_auth.py:ro
And in config.yaml, under general_settings:
custom_auth: jwt_auth.user_api_key_auth
custom_auth_run_common_checks: true
That second line matters more than it looks. Without it, LiteLLM warns at startup:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-custom-auth-warning-863f445ce677a95c6cf806de7ecfc4c7.png\")
\n\n\"custom_auth is configured but 'custom_auth_run_common_checks' is not set. Problem: budgets,\nmodel-access allowlists, and per-model rate limits configured on your DB team/project records\nwill NOT be enforced for custom-auth requests.\"
\n
Read that carefully. Adding token authentication would, by default, have exempted every token\nholder from the controls you already configured. You would have made the gateway more secure in\none dimension and quietly weaker in another, and the only notice is one warning line at\nstartup.
\nconfig.yaml is a bind mount. Changing its contents changes no part of the compose spec, so\ndocker compose up -d reports Container llm-gateway Running and does nothing. Your tests\nthen run against the old process and reproduce the old results exactly, which reads like your\nchange had no effect.
Use docker restart llm-gateway, and confirm by the warning above disappearing from the log.
TOKEN=$(~/agent-auth/token.sh gateway:invoke)
NOSCOPE=$(~/agent-auth/token.sh)
curl -sS -o /dev/null -w 'no token HTTP %{http_code}\\n' http://127.0.0.1:4000/v1/chat/completions \\
-H 'Content-Type: application/json' \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Say OK.\"}]}'
curl -sS -o /dev/null -w 'wrong scope HTTP %{http_code}\\n' http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $NOSCOPE\" -H 'Content-Type: application/json' \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Say OK.\"}]}'
curl -sS -o /dev/null -w 'valid token HTTP %{http_code}\\n' http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $TOKEN\" -H 'Content-Type: application/json' \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Say OK.\"}]}'
no token HTTP 401
wrong scope HTTP 401
valid token HTTP 429
![\"Three](\"https://development-wec.wiline.com/docs/assets/images/h5-three-status-codes-0db858ce4b80dda45f7c2626ec09529c.png\")
The first two never reach a model. The second is the one that matters: a valid, correctly\nsigned, unexpired token from the right issuer, refused because it did not carry\ngateway:invoke. Identity alone is not authority.
The third is the interesting one, and it is not a 200. It cleared authentication and scope, and\nwas then answered by a completely different part of the gateway ā the budget and routing layer\nthe gateway series configures, which on this box\nhad a cap already reached. That is the point of the shape rather than a flaw in it: a 401 means\nwe do not know who you are, or you may not do this, and anything else means the request got\npast identity entirely and is now somebody else's decision. Authentication's job ends at the\nfirst two lines.
\nConfirm the refusal came from your check rather than something incidental:
\ndocker logs llm-gateway --since 10m 2>&1 | grep -iE 'missing scope|401 Unauthorized' | tail -5
user_api_key_auth(): Exception occured - token is missing scope gateway:invoke
Exception: token is missing scope gateway:invoke
INFO: \"POST /v1/chat/completions HTTP/1.1\" 401 Unauthorized
![\"The](\"https://development-wec.wiline.com/docs/assets/images/h5-scope-refusal-log-aec61cc01ab970a987dee6c534e39c59.png\")
Note where that reason lives. The caller receives a bare 401 with no explanation, which is\ncorrect ā telling a client \"your token is valid but lacks scope X\" tells an attacker exactly\nwhat to go and get. It does mean that debugging this always means reading the gateway log, and\nnever the response body.
\nYou can give a headless agent its own identity: a confidential client restricted to one grant,\na scope that means something, and a gateway that verifies the signature locally and refuses\nanything without the scope ā with the refusal proven from the log rather than assumed from a\nstatus code.
invalid_clientāThe credentials did not authenticate at all. Before blaming authentik, check what actually\nlanded in your .env:
awk -F= '{printf \"%-18s %3d chars\\n\", $1, length($2)}' ~/agent-auth/.env
AK_TOKEN_URL 48 chars
AK_CLIENT_ID 40 chars
AK_CLIENT_SECRET 128 chars
The client ID is 40 characters and the secret is 128. A zero-length value means the file is the\nproblem, not the provider.
\ninvalid_grant, with everything apparently correctāThe provider is not paired to an application. See Step 2 ā the Providers list flags it, the\nerror text does not.
\nAK_TOKEN_URL: unbound variable, and a corrupted secretāecho 'X=1' >> .env on a file whose last line has no trailing newline appends to that line\ninstead of creating a new one:
AK_CLIENT_SECRET=p94oDU...dgJcAK_TOKEN_URL=http://10.80.4.212:9100/application/o/token/
Two variables destroyed in one command, silently ā the secret is now wrong and the new\nvariable does not exist. Check with cat -A, which marks line ends with $, and prefer\nwriting the whole file with a single heredoc over appending to it.
The container was never restarted. See the warning in Step 4.
\n401 with no reason in the responseāBy design. The reason is in docker logs llm-gateway.
Done: a machine identity issued by your identity provider rather than minted by hand, a\ncredential that expires in five minutes, a scope that has to be requested and is checked on\nevery call, revocation that happens in one place, and a caller the gateway can name in its logs\nas something other than a key prefix.
\nNot done:
\nissue_refund included. Same problem, one layer up, and the subject of\nthe next part.scope_mappings between scopes and models is where that goes.ak-agent-gateway-client_credentials without asking, and nothing in the token says it is a\nmachine rather than a person.The MCP server. FastMCP ships a JWTVerifier that takes a jwks_uri, an issuer, an\naudience and required_scopes ā the same four things configured here ā so the tools server\ncan demand a scoped token before it will list a tool, let alone run one. That closes the gap\nagent orchestration Part 4 left open, where scoping happened in the client and the server\ntrusted whoever reached the port.
PyJWKClientPart 3 gave an agent five tools over MCP ā\na calendar, a customer list, invoices, and a refund ā and put a human in front of the refund.\nIt ended by admitting the obvious: one agent held all five. Nothing stopped the model\nreaching for issue_refund when it had been asked to book an appointment. The human pause was\nthe only thing in the way, and a pause only fires if the model calls the tool at all.
This part removes the tool instead. The scheduler gets three tools and the billing agent gets\nthree, sharing one. Ask the scheduler to refund an invoice and it cannot ā not because it was\ntold not to, not because a policy blocked it, but because issue_refund was never in the list\nit was given.
Then Part 2's supervisor goes back on top,\nand the interesting property falls out: routing decides who works, scoping decides what is\npossible. A misrouted refund still cannot refund.
\n\nThe same box, server and database as Part 3:\none WEC Instance, Ubuntu 22.04, Python 3.10, langchain 1.4.0, langgraph 1.2.11, mcp 2.1.1,\nfastmcp 4.0.2. office_tools.py unchanged and still serving five tools on\n127.0.0.1:8770. Models through the LiteLLM gateway to WEC Inference. Reset the database with\n./.venv/bin/python seed.py before following along, so invoice 2 is open again.
There is no new mechanism in this post. create_agent takes a list of tools; we hand it a\nshorter list. That is the entire technique, and it is worth being precise about why it is\nstronger than the alternatives:
The last one is the only one that doesn't depend on inference going well.
\nThe server offers all five to anyone who asks. What differs is which ones each agent is\nhanded. find_customer goes to both, because looking a customer up is harmless.
The blocked arrow is the whole post. The scheduler cannot call issue_refund ā not because a\nrule forbids it, but because the tool is not in the list it was given, so it never reaches the\nmodel's context at all.
~/mcp-tools/.env, as Part 3 set upPart 3 ran the server in the foreground, which cost a terminal and died with the SSH session.\nStart it detached instead ā one session is enough for everything below:
\ncd ~/mcp-tools && nohup ./.venv/bin/python office_tools.py > server.log 2>&1 & sleep 3; ss -tlnp | grep 8770
The server still offers five tools. The roles name subsets of them:
\nimport asyncio, os, sys
from langchain.agents import create_agent
from langchain.mcp import MCPAdapter
from langchain_openai import ChatOpenAI
from langgraph.checkpoint.memory import InMemorySaver
MCP_URL = \"http://127.0.0.1:8770/mcp\"
ROLES = {
\"scheduler\": (\"find_customer\", \"list_availability\", \"book_slot\"),
\"billing\": (\"find_customer\", \"open_invoices\", \"issue_refund\"),
}
model = ChatOpenAI(
base_url=\"http://127.0.0.1:4000/v1\",
api_key=os.environ[\"GATEWAY_KEY\"],
model=\"qwen-mid\",
temperature=0,
)
async def main():
role, request = sys.argv[1], sys.argv[2]
async with MCPAdapter(MCP_URL) as adapter:
catalog = {t.name: t for t in await adapter.list_tools()}
print(f\"server offers : {sorted(catalog)}\")
allowed = [catalog[n] for n in ROLES[role]]
print(f\"{role} can see : {[t.name for t in allowed]}\\n\")
agent = create_agent(model, allowed, checkpointer=InMemorySaver())
cfg = {\"configurable\": {\"thread_id\": f\"{role}-1\"}}
out = await agent.ainvoke({\"messages\": [{\"role\": \"user\", \"content\": request}]}, cfg)
if \"__interrupt__\" in out:
req = out[\"__interrupt__\"][0].value[\"requests\"][0]
print(f\"PAUSED ā the server is asking a human:\\n {req['message']}\")
return
print(\"--- answer ---\")
print(out[\"messages\"][-1].content)
asyncio.run(main())
The checkpointer is carried over from\nPart 3: issue_refund pauses the run with an\ninterrupt, and an interrupted run needs somewhere to wait. The scheduler never triggers one, but\nboth roles run the same code.
The whole of the technique is one line:
\nallowed = [catalog[n] for n in ROLES[role]]
find_customer appears in both roles ā looking a customer up is harmless and both jobs need\nit. book_slot and issue_refund appear in exactly one each.
The script prints both lists on every run for a reason. server offers is the full catalog the\nMCP server returned; can see is what we passed to create_agent. Keeping them side by side\nmakes it obvious that the server hid nothing ā the narrowing happened in the client, on the\nline above.
Ask the billing agent to do its job:
\ncd ~/mcp-tools && set -a && source .env && set +a && ./.venv/bin/python scoped.py billing \"Refund Maria's open invoice.\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg4-billing-paused-6de8f7f7d8c481db7073918444d88e53.png\")
It found Maria, resolved \"open invoice\" to invoice 2, called issue_refund, and stopped ā\nbecause the tool demands a human. That is Part 3's guard, working.
Now the identical sentence to the wrong specialist:
\ncd ~/mcp-tools && ./.venv/bin/python scoped.py scheduler \"Refund Maria's open invoice.\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg4-scheduler-refused-080822f69c551c7655e8e5fa5093178b.png\")
\n\nI don't have access to tools that can process refunds or handle invoices. The available tools\nI have are for finding customers, listing appointment availability, and booking appointment\nslots.
\n
That answer is accurate and well-mannered, and it would be a mistake to read it as the model\nbehaving well. The model could not have called issue_refund. It was never in the list, so\nit was never in the context, so there was no call to decline.
The distinction matters because a model that does hold a dangerous tool can also produce a\npolite sentence ā including one describing an action it never took. We documented exactly that\nin the news piece: an era error handed to the model, and the\nmodel narrating an approval nobody would ever be asked for. A refusal you can trust is one the\nmodel had no way around.
And the scheduler doing the job it does have tools for, so the restriction is targeted rather\nthan crippling:
\ncd ~/mcp-tools && ./.venv/bin/python scoped.py scheduler \"Book Maria into a free slot on 2026-09-05\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg4-scheduler-books-56592bddc8b3d38f6f2f0f8d8aaf64ec.png\")
A refusal is a sentence. The database is the evidence.
\ncd ~/mcp-tools && ./.venv/bin/python -c \"
import sqlite3
for r in sqlite3.connect('office.db').execute('select id, customer_id, amount, status from invoices'): print(r)\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg4-nothing-happened-3c326c0194b72bcc27c1a87d9a2d05f7.png\")
Invoice 2 is still open, for two different reasons: the scheduler had no tool to change it,\nand billing was stopped by the human gate before it could. Neither depended on the model\nchoosing well.
It is worth seeing the same agent report a boring truth, too. Before we reset the database,\ninvoice 2 had already been refunded in Part 3 ā and asked to refund it again, billing looked,\nfound nothing open, and said so:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg4-billing-nothing-open-b4f526ef9e6865039b0aefe2909f7918.png\")
No invented invoice, no cheerful confirmation. That is the behaviour that makes its other\nanswers worth anything, and it is not something to take on trust ā it is something to check,\nwhich is why every claim in this post has a query behind it.
\nYou can give two agents different slices of the same MCP server, so that a capability one of\nthem must never have is absent from its context rather than forbidden by instruction ā and\nverify in the data that the absence held.
Part 2 routed work between a scheduler and a billing agent that had no tools. Now they do.
\nsupervisor.py is scoped.py with Part 2's graph around it: the same ROLES, MCPAdapter and\nChatOpenAI setup, plus a State TypedDict, a StateGraph, and one agent built per role\ninside the async with block ā agents = {role: create_agent(model, [catalog[n] for n in names]) for role, names in ROLES.items()}. Only the parts that are new to this post are shown here.
BILLING_WORDS = (\"invoice\", \"refund\", \"bill\", \"charge\", \"payment\")
def supervisor(state: State) -> Command[Literal[\"scheduler\", \"billing\"]]:
nxt = \"billing\" if any(w in state[\"request\"].lower() for w in BILLING_WORDS) else \"scheduler\"
print(f\"supervisor routes to {nxt}\")
return Command(goto=nxt, update={\"completed\": [f\"supervisor->{nxt}\"]})
async def run(role: str, state: State):
out = await agents[role].ainvoke(
{\"messages\": [{\"role\": \"user\", \"content\": state[\"request\"]}]})
if \"__interrupt__\" in out:
q = out[\"__interrupt__\"][0].value[\"requests\"][0][\"message\"]
return {\"answer\": f\"PAUSED ā {q}\", \"completed\": [role]}
return {\"answer\": out[\"messages\"][-1].content, \"completed\": [role]}
async def scheduler_node(state: State):
return await run(\"scheduler\", state)
async def billing_node(state: State):
return await run(\"billing\", state)
The Command[Literal[\"scheduler\", \"billing\"]] annotation is doing the same work it did in Part\n2 ā it is the only declaration of where the supervisor may send work, and dropping it makes\nevery graph diagram lie.
cd ~/mcp-tools && set -a && source .env && set +a && ./.venv/bin/python supervisor.py \"Refund Maria's open invoice.\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg4-supervisor-refund-eff231a6a04d02fc5430fa25025750e6.png\")
cd ~/mcp-tools && ./.venv/bin/python supervisor.py \"Book Maria into a free slot on 2026-09-04\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg4-supervisor-book-cb9829d31562e18281b2165e7fb3bf6e.png\")
That supervisor is a keyword match. It is trivially foolable ā \"settle Maria's outstanding\namount\" contains none of BILLING_WORDS and lands on the scheduler.
Which is the point worth taking away. A misrouted refund still cannot refund. The router\ndecides who gets the work; the toolsets decide what that agent can do with it. If routing were\nthe only control, every bug in the supervisor would be a security bug. With scoping underneath,\na routing mistake is just a bad answer.
You can put a router in front of scoped specialists and reason clearly about which failures are\ndangerous ā because the blast radius of a routing bug is bounded by what the receiving agent\nwas given, not by what it was asked to do.
InvalidUpdateError: Expected dict, got <coroutine object>āThe node is an async function wrapped in a lambda:
builder.add_node(\"scheduler\", lambda s: run(\"scheduler\", s)) # returns a coroutine
LangGraph calls it, gets a coroutine back instead of a state update, and raises. The real clue\nis the last line of the output ā RuntimeWarning: coroutine 'run' was never awaited ā which is\nthe part nobody reads. Define proper async def nodes instead.
RuntimeError: Client failed to connect: All connection attempts failedāNothing is listening on 8770. If you started the MCP server in a foreground shell, it died with\nyour SSH session. The error names the client, which sends you looking in the wrong place.
\nss -tlnp | grep 8770
Start it with nohup ... & so it survives, and check the port before blaming the code.
Python buffers stdout when it is redirected to a file, so a long run looks identical to a hang.\nAdd -u:
nohup ./.venv/bin/python -u supervisor.py \"...\" > run.log 2>&1 &
tail -f run.log
KeyError: 'GATEWAY_KEY'āsource .env sets shell variables, not environment variables, and export does not cross SSH\nsessions. Every command that starts the agent needs its own\nset -a && source .env && set +a ā including the backgrounded ones.
Done: two agents drawing different subsets of one MCP server, a capability that is absent\nrather than forbidden, and a router whose mistakes cost correctness instead of money.
\nNot done:
\n127.0.0.1:8770 can call issue_refund directly, agent or not. This is a control on\nwhat your agents can do, not on what your server will accept.Push the boundary from the client to the server: authenticate the tool call, so the server\nknows which agent is asking and refuses issue_refund to anything that isn't billing. That is\nthe same question Part 4 of the hardening series\nasked about the gateway's own API ā a control plane and a data plane, one layer apart ā and\nFastMCP already supports bearer tokens and OAuth 2.1 for it.
Command and conditional edges2026-07-28 specificationPart 3 put authentik in front of Langfuse and\nended with an admission: the bindings step was left empty, so every account in authentik\ncould reach Langfuse and nothing would warn you.
\nThis part closes that gap on the LiteLLM gateway\nā and the closing went wrong in a way worth more than the procedure. We configured the\nbindings in authentik's wizard, watched them appear in the wizard's own table, submitted,\nand ended up with zero bindings saved. The application was open to everyone, the UI said\nnothing, and the only reason we found out is that we tested with an account that should have\nbeen refused and wasn't.
\nThen, once access control genuinely worked, we called the gateway's API from a shell with no\naccount, no session and no group. It answered normally. That one isn't a bug ā it's the\ndifference between a control plane and a data plane, and assuming SSO covers both is\nhow people conclude they've secured something they haven't.
\n\nOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Docker 29.1.3, kernel 5.15.\nauthentik 2026.8.1, the same instance from Part 3 on host port 9100. LiteLLM\nmain-stable from Part 1 of the gateway series,\nbound to 127.0.0.1:4000, models served over WEC Inference.
Plain HTTP on a private LAN, as in Part 3.
Part 3 taught the wizard ā application, provider, redirect URI, scopes. That isn't repeated;\nit's all in Part 3 and the steps are identical for\nany app. Three things are new:
\nA group, as the unit you grant access to, so that adding somebody to a team and giving\nthem access are the same action.
\nA binding, which attaches that group to an application, so reaching it requires\nmembership rather than merely existing.
\nA boundary, demonstrated rather than asserted: what SSO protects, and what it doesn't\ntouch at all.
\nLITELLM_MASTER_KEY, which you'll need to create a virtual key127.0.0.1Two boxes, no arrow between them ā that's the point. Everything this post configures lives\nin the left one. The right one never consults authentik at all.
\nThere is also a third way in that the diagram deliberately leaves out, because it belongs to\nneither plane: LiteLLM's master key logs into the Admin UI directly, skipping the group\ncheck entirely. We hit it in Step 5.
\nDirectory ā Groups ā New Group, name it gateway-admins, and leave Superuser\nprivileges off. That flag makes members administrators of authentik itself and has\nnothing to do with which applications they can reach.
authentik ships three groups already ā authentik Admins, authentik Agent-Users,\nauthentik Read-only ā and authentik Admins already contains your akadmin. Binding that\none instead would work today and quietly mean \"anyone I ever make an authentik admin,\" which\nis not the same intent.
![\"authentik's](\"https://development-wec.wiline.com/docs/assets/images/ak-groups-list-a17357dbbd4468da8f7232d5487ef77a.png\")
Add akadmin to gateway-admins. Then Directory ā Users ā New User, type Internal,\nusername contractor, and no groups at all.
Internal rather than External so that the only difference between the two accounts is\nmembership ā any other difference muddies the result. Not a Service Account: those are\nmachine-to-machine and don't do interactive logins.
\nauthentik creates users without a password, and the Edit dialog has no password field ā it's\na separate action on the user's detail page. From the shell:
\ncd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import User
u = User.objects.get(username='contractor')
u.set_password('Contractor123!')
u.save()
print('password set for', u.username)
\" 2>/dev/null | tail -2
ak shellEvery ak shell call prints roughly sixty lines of bootstrap JSON before your output.\n2>/dev/null | tail -N drops the noise and keeps the last N lines. Used throughout below.
Run the Part 3 wizard: application LiteLLM, provider OAuth2/OpenID, authorization\nflow default-provider-authorization-explicit-consent, client type Confidential.
The gateway is published on 127.0.0.1:4000, so your browser can't reach it ā and an OIDC\nlogin happens in a browser. Open a tunnel from your workstation:
ssh -L 4000:127.0.0.1:4000 ubuntu@10.80.4.212
ssh -L warns, then carries on without the tunnelIf something already holds local port 4000, ssh prints bind: Address already in use and\nstill gives you a working shell ā with a dead forward. The only symptom is\nERR_CONNECTION_REFUSED in the browser, which reads like the gateway being down. Check the\nport before blaming the gateway: lsof -nP -iTCP:4000 -sTCP:LISTEN.
Everything the browser types is now localhost:4000, so that's the redirect URI, mode\nStrict:
http://localhost:4000/sso/callback
Now the first surprise. Ask authentik what it actually named things:
\ncd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import Application
for a in Application.objects.all():
print(repr(a.name), '->', repr(a.slug))
\" 2>/dev/null | tail -5
'Langfuse' -> 'langfuse'
'LiteLLM' -> 'lite-llm'
lite-llm, not litellm. The slugifier read the capitals in \"LiteLLM\" and split them.\nPart 3 called the slug load-bearing because it lands in the issuer URL, and here it is:\nhttp://10.80.4.212:9100/application/o/lite-llm/. Set the slug explicitly rather than\nletting a CamelCase name generate one, or carry that hyphen in every URL forever.
Read the credentials back through the real slug:
\ncd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import Application
p = Application.objects.get(slug='lite-llm').get_provider()
print('CLIENT_ID :', p.client_id)
print('CLIENT_SECRET:', p.client_secret)
\" 2>/dev/null | tail -3
The wizard's step 4 is Configure Bindings, and Part 3 walked past it showing No bound policies. This time we didn't walk past it. We bound the group there, the wizard listed it\nin its table, and we submitted.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/ak-bindings-wizard-empty-996a2adb20faf53dda84769592740e1c.png\")
![\"The](\"https://development-wec.wiline.com/docs/assets/images/ak-bindings-wizard-group-2c88ba715a495db175a3e85cf83df1f9.png\")
Then we asked the database what existed:
\ncd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import Application
app = Application.objects.get(slug='lite-llm')
bs = list(app.bindings.all())
print('binding count:', len(bs))
for b in bs:
print('group', b.group, '| enabled', b.enabled, '| negate', b.negate)
\" 2>/dev/null | tail -4
binding count: 0
Zero. The application's own bindings tab agreed ā No Policies bound. Two independent\nsources, one conclusion: the wizard displayed access control that was never written.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/ak-bindings-none-71e30bfc7aa65339a119ee1ccb38d3ab.png\")
Part 3's lesson was \"don't skip bindings.\" The real hazard is that you can complete them,\nsee your rules in the wizard's table, submit, and end up with an application that admits\neveryone ā no error, no warning, and a UI that looked correct the whole time.
An application with zero bindings isn't broken in any way authentik reports. It's doing\nexactly what it should: nothing bound means nobody is excluded. Verify bindings from the\ndatabase or the application's own bindings tab ā never from the wizard.
Bind it from the application instead: Applications ā LiteLLM ā Policy / Group / User\nBindings ā Create or bindā¦, then Bind a group under Bind Existingā¦ ā gateway-admins\nā Create.
Pick Bind a group specifically. The Choose Policy Type section below it creates\nbrand-new policies (Event Matcher, Expression, GeoIP); a group binding is already a rule and\nneeds none of them.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/ak-bind-group-dialog-c20ad52e0f590a4df1c9a5e3aa411963.png\")
Then verify, because the table is what lied:
\nbinding count: 1
group Group gateway-admins | enabled True | negate False
You can restrict an application to a group instead of leaving it open to every account ā\nand, more importantly, verify the binding actually persisted (app.bindings.all()) rather\nthan trusting the wizard's own table.
That page states the mode plainly: \"The currently selected policy engine mode is ANY: Any\npolicy must match to grant access.\" With one binding, ANY and ALL behave identically. With\ntwo they don't ā ANY means one passing rule is enough, so a stray User Contractor\nbinding sitting beside the group would grant precisely the access this post is trying to\ndeny.
LiteLLM reads generic OIDC settings from GENERIC_*. The endpoints come from the discovery\ndocument:
cat >> ~/llm-gateway/.env <<'EOF'
GENERIC_CLIENT_ID=<client id from Step 2>
GENERIC_CLIENT_SECRET=<client secret from Step 2>
GENERIC_AUTHORIZATION_ENDPOINT=http://10.80.4.212:9100/application/o/authorize/
GENERIC_TOKEN_ENDPOINT=http://10.80.4.212:9100/application/o/token/
GENERIC_USERINFO_ENDPOINT=http://10.80.4.212:9100/application/o/userinfo/
PROXY_BASE_URL=http://localhost:4000
EOF
PROXY_BASE_URL is the one that bites. LiteLLM builds its redirect URI from it, so it has to\nequal both what the browser types and what authentik has registered. Ours disagreed on the\nfirst attempt and produced a Redirect URI Error ā the redirect_uri in the address bar\nis what LiteLLM sent, the Strict value on the provider is what authentik expected, and a\nport, a hostname or a trailing slash between them is enough to fail.
The gateway's compose file uses env_file: .env, so unlike Langfuse in Part 3 there's no\noverride file to write:
cd ~/llm-gateway && docker compose up -d && docker compose exec litellm env | grep -c GENERIC_
Five. Before the edit it was zero ā the same class of check Part 3 used to catch variables\nthat never reached the process.
\nOpen http://localhost:4000/ui through the tunnel. The login page carries a finding this\npost didn't go looking for.
Login with SSO appears, as intended. Above it sits a username and password form, and\nLiteLLM's own panel explains it: \"By default, Username is admin and Password is your set\nLiteLLM Proxy MASTER_KEY.\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/litellm-login-sso-a0616071a645fb27014bfa0b35203eaa.png\")
The master key is a complete bypass. Anyone holding it reaches the admin UI without touching\nauthentik, without a group, without a binding. Everything configured in Steps 1ā3 sits beside\na door that ignores all of it ā and that door's key is also the gateway's root credential,\nable to mint API keys and reach every model.
\nAUTO_REDIRECT_UI_LOGIN_TO_SSO=true sends /ui straight to authentik instead of showing the\nform. It stops advertising the password path; it doesn't remove it. Treat the master key as\na break-glass credential and don't mistake the redirect for a fix.
Sign in as akadmin, who is in gateway-admins. authentik shows the consent screen from\nPart 3's explicit-consent flow, and you land in the dashboard at /ui/?login=success.
![\"authentik's](\"https://development-wec.wiline.com/docs/assets/images/ak-login-litellm-9e4c240eccffd4f7e739199f9b9c594a.png\")
![\"The](\"https://development-wec.wiline.com/docs/assets/images/ak-consent-akadmin-ea2ed59007a2b3e896f2d0437553763f.png\")
Now contractor.
Opening \"a new private window\" while another is already open reuses the same cookies, so\nauthentik signs you back in as the previous user. The symptom is an instant login that looks\nlike your access control failed. Close every incognito window, or use a different\nbrowser.
Permission denied. Request has been denied.
\n![\"authentik](\"https://development-wec.wiline.com/docs/assets/images/ak-contractor-denied-14fb1a97964d42b993f3e49504ecb9dd.png\")
Note where that happened: authentik refused before the consent screen. The policy check\nruns ahead of consent, so a blocked user never sees the permissions page ā which is also how\nwe caught the missing bindings earlier. When contractor reached a consent screen, that was\nthe tell.
From a shell on the box ā no authentik account, no session, no group, nothing but a key:
\ncurl -s http://127.0.0.1:4000/v1/chat/completions -H \"Authorization: Bearer sk-...\" -H \"Content-Type: application/json\" -d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Say OK.\"}]}' | jq '{model, answer: .choices[0].message.content}'
{
\"model\": \"qwen-mid\",
\"answer\": \"\\n\\nOK\"
}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/litellm-api-still-open-f3c81e08b6e2981f5cd844d8f236a4ec.png\")
Nothing is broken. SSO never applied to that endpoint. Steps 1ā5 protected /ui ā the\ncontrol plane, where humans create keys and read spend. The data plane at /v1/*\nauthenticates machine callers with API keys, and it has to: an agent running at 3am can't\ncomplete a browser login with a consent screen.
That key was itself created from a shell, with the master key, while no SSO session existed\nanywhere:
\ncd ~/llm-gateway && source .env && curl -s http://127.0.0.1:4000/key/generate -H \"Authorization: Bearer $LITELLM_MASTER_KEY\" -H \"Content-Type: application/json\" -d '{\"key_alias\":\"sso-demo\",\"models\":[]}' | jq -r '.key'
The consequences, stated plainly, because \"we put SSO on the gateway\" gets said in meetings\nas though it settled the question:
\ngateway-admins does not revoke their keys. They lose the UI and\nkeep every key they made. Offboarding is two actions; only one is the one people remember.The gateway's own key model ā scoped keys, budgets, expiry, per-key models ā is what covers\nthat ground, and it's a different mechanism from the one configured here. Complementary, not\nsubstitutable.
\nSigned in as akadmin through SSO, the Virtual Keys page had no Create Key button.
LiteLLM assigns SSO users a default role that isn't admin. Passing authentik's binding got us\ninto the UI; what we could do there is LiteLLM's own role model, which we never\nconfigured. Authentication, then authorization, then application-level roles ā three distinct\nlayers, and clearing one says nothing about the next.
\nApplication matching query does not existāThe ak shell lookup fails even though the application is visibly there in the UI. The slug\nisn't what you assumed ā authentik derives it from the name, and a CamelCase name like\nLiteLLM becomes lite-llm. Ask rather than guess:
cd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import Application
for a in Application.objects.all():
print(repr(a.name), '->', repr(a.slug))
\" 2>/dev/null | tail -5
authentik refuses before any login form. The redirect_uri in the address bar is what\nLiteLLM sent; the Strict value on the provider is what authentik expects. Ours disagreed\nbecause PROXY_BASE_URL said one host and the registered URI said another. Print what's\nregistered and compare character by character:
cd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import Application
p = Application.objects.get(slug='lite-llm').get_provider()
for r in p.redirect_uris:
print(repr(r.matching_mode), repr(r.url))
\" 2>/dev/null | tail -5
If you edit it and the error persists, Applications ā Clear cache ā authentik caches\nprovider config.
\nERR_CONNECTION_REFUSED on localhost:4000āThe tunnel isn't forwarding, even though ssh connected. If something already held the local\nport, ssh printed bind: Address already in use and carried on with a working shell and a\ndead forward. Check the port, then restart the tunnel:
lsof -nP -iTCP:4000 -sTCP:LISTEN
Two causes, both real here. Either a stray user binding grants them directly ā policy\nengine mode ANY means one passing rule is enough ā or the bindings were never saved. Ask\nthe database, not the UI:
\ncd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import Application
app = Application.objects.get(slug='lite-llm')
bs = list(app.bindings.all())
print('binding count:', len(bs))
for b in bs:
print('group', b.group, '| user', b.user, '| enabled', b.enabled, '| negate', b.negate)
\" 2>/dev/null | tail -5
A user who reaches the consent screen has already passed the policy check ā that's the\ntell. A blocked user never gets that far.
\nBrowser session, not policy. All Chrome incognito windows share one session, so opening\n\"a new private window\" while another is open keeps the previous user signed in. Close every\nincognito window, or use a different browser.
\nYou're signed in through SSO, and LiteLLM gives SSO users a default role that isn't admin.\nCreate keys from the shell with the master key, or configure LiteLLM's own roles ā a\nseparate mechanism from anything authentik does.
\nYou can tell a control plane from a data plane on your own stack: put SSO and group\nmembership in front of an admin UI, then prove the API behind it still answers on a key\nalone ā and name what that means for offboarding, leaked keys and keys that outlive people.
Done: a group that means something, a binding that enforces it and that we verified in the\ndatabase rather than trusting a table, and an admin UI where access follows membership.
\nNot done:
\nTLS and real exposure. The gateway is on 127.0.0.1:4000 and reachable in this post only\nthrough an SSH tunnel; authentik is plain HTTP. Making either properly reachable means a\nreverse proxy, a hostname and a certificate ā and every redirect URI, PROXY_BASE_URL and\nissuer changes the day that happens.
Part 1 gave an agent state that survives a\nrestart, and a pause where a human approves before it continues. Part 2\nput a supervisor in front of a scheduler and a billing agent and routed work between\nthem.
\nNeither of those agents scheduled or billed anything. They discussed it. Every tool they had\nwas a function you wrote into the same process, and the interesting ones ā look up a customer,\ntake a slot, move money ā didn't exist.
\nThis part builds the toolbox those two roles need, over MCP: a calendar, a customer list,\ninvoices, and a refund. To keep the moving parts down we point one agent at all five tools\nrather than rebuilding Part 2's supervisor ā the agent discovers them at runtime instead of\nbeing wired to them, chains them to answer a question, and writes rows you can go and check in\nthe database afterwards. Then the refund stops it dead and makes a human type the amount.
\nSplitting these tools back across the scheduler and the billing agent ā so the scheduler\ncannot see issue_refund at all ā is the natural next step, and the last section says how.
In July the MCP specification dropped sessions entirely. On the\nday we ran this, LangChain shipped support for that revision in the main package. So this is a\nfirst run against a five-week-old protocol and a same-day client ā which is why three things\nin here aren't in anyone's documentation yet.
\n\nOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Ubuntu 22.04, Python 3.10. langchain\n1.4.0, langchain-openai 1.6.0, langgraph 1.2.11, mcp 2.1.1, fastmcp 4.0.2 ā all\ninstalled on the day of writing. Models served through the LiteLLM gateway from\nPart 1 of the gateway series, which forwards to\nWEC Inference. langchain.mcp is in beta and warns on every import; the API may move.
A server publishes tools. A client asks it what tools exist. The model picks one and the\nclient calls it. That's the whole protocol ā the value is that \"asks what tools exist\" is\nstandardised, so any client can use any server without glue written for the pair.
\nThree words you need:
\nTool. A function the server exposes, with a description and a JSON schema for its\narguments. The description is what the model reads to decide whether to call it; the schema is\nwhat it must fill in. Nothing else about your code reaches the model.
\nDiscovery. The client asks tools/list and gets that catalog back. Your agent learns what\nit can do at runtime instead of at the time you wrote it.
Elicitation. A tool that can't finish without asking a human something ā confirming a\ndelete, supplying a parameter the model left out. Under the new spec this is an ordinary\nrequest the client retries with an answer attached.
\nUnder the old spec a tool that needed input held the connection open and asked down a\nback-channel. Under 2026-07-28 it returns, and the client calls it again with the answer\nattached. Two complete requests instead of one long-lived one:
The tool runs twice. That's why its body starts with if ctx.input_responses is None ā it\nhas to be written to be re-entered, not resumed. Nothing is held open in between, so the answer\ncan arrive after a redeploy, or at a different replica behind a load balancer.
MCP support now lives in the main langchain package. If you have langchain-mcp-adapters\ninstalled from before, it is the thing this replaces.
mkdir -p ~/mcp-tools && cd ~/mcp-tools && python3 -m venv .venv && ./.venv/bin/pip install -q \"langchain[mcp]\" langchain-openai httpx && ./.venv/bin/pip list | grep -iE 'langchain|langgraph|^mcp |fastmcp'
![\"pip](\"https://development-wec.wiline.com/docs/assets/images/mcp-versions-5b3443d87f1bcddd968257037aceb26f.png\")
langchain[mcp] needs 1.4.0 or newer ā below that, MCPAdapter doesn't exist.
mcp[cli] is goneOlder guides tell you to pip install \"mcp[cli]\". On mcp 2.x that extra no longer exists,\nand pip fails with a wall of does not provide the extra 'cli' across sixty versions before\ngiving up with ResolutionImpossible. It reads like a dependency conflict; it's a removed\nextra.
The agents need data that exists. SQLite, seeded once ā three customers, six slots (one\nalready taken), four invoices.
\nimport sqlite3
db = sqlite3.connect(\"office.db\")
db.executescript(\"\"\"
DROP TABLE IF EXISTS customers; DROP TABLE IF EXISTS slots; DROP TABLE IF EXISTS invoices;
CREATE TABLE customers(id INTEGER PRIMARY KEY, name TEXT, email TEXT);
CREATE TABLE slots(id INTEGER PRIMARY KEY, day TEXT, time TEXT, customer_id INTEGER);
CREATE TABLE invoices(id INTEGER PRIMARY KEY, customer_id INTEGER, amount REAL, status TEXT);
INSERT INTO customers(name,email) VALUES
('Maria Alvarez','maria@example.com'),
('Tomas Reis','tomas@example.com'),
('Priya Nair','priya@example.com');
INSERT INTO slots(day,time,customer_id) VALUES
('2026-09-04','09:00',NULL),('2026-09-04','11:00',NULL),('2026-09-04','14:00',2),
('2026-09-05','09:00',NULL),('2026-09-05','11:00',NULL),('2026-09-05','16:00',NULL);
INSERT INTO invoices(customer_id,amount,status) VALUES
(1,240.00,'paid'),(1,80.00,'open'),(2,150.00,'paid'),(3,410.00,'open');
\"\"\")
db.commit()
Two details are load-bearing. Maria has both a paid and an open invoice, so \"refund Maria's\ninvoice\" is ambiguous and the agent has to resolve it. Tomas already holds Friday at 14:00,\nso availability is a real question rather than \"everything\".
\nFive tools: three reads, one write, one that cannot finish without a human.
\nimport os, sqlite3
from fastmcp import FastMCP, Context
from mcp.types import InputRequiredResult, ElicitRequest, ElicitRequestFormParams
DB = os.path.join(os.path.dirname(os.path.abspath(__file__)), \"office.db\")
mcp = FastMCP(\"office-tools\")
def q(sql, args=()):
con = sqlite3.connect(DB); con.row_factory = sqlite3.Row
try:
rows = con.execute(sql, args).fetchall(); con.commit()
return [dict(r) for r in rows]
finally:
con.close()
def w(sql, args=()):
con = sqlite3.connect(DB)
try:
cur = con.execute(sql, args); con.commit(); return cur.rowcount
finally:
con.close()
@mcp.tool
async def find_customer(query: str) -> list[dict]:
\"\"\"Find customers whose name or email matches the query.\"\"\"
like = f\"%{query}%\"
return q(\"SELECT id, name, email FROM customers WHERE name LIKE ? OR email LIKE ?\", (like, like))
@mcp.tool
async def list_availability(day: str) -> list[dict]:
\"\"\"List free appointment slots on a given day, formatted YYYY-MM-DD.\"\"\"
return q(\"SELECT id, day, time FROM slots WHERE day = ? AND customer_id IS NULL ORDER BY time\", (day,))
@mcp.tool
async def book_slot(slot_id: int, customer_id: int) -> str:
\"\"\"Book a free slot for a customer. Fails if the slot is already taken.\"\"\"
n = w(\"UPDATE slots SET customer_id = ? WHERE id = ? AND customer_id IS NULL\", (customer_id, slot_id))
if n == 0:
return f\"Slot {slot_id} is already taken or does not exist.\"
row = q(\"SELECT day, time FROM slots WHERE id = ?\", (slot_id,))[0]
return f\"Booked slot {slot_id} ({row['day']} {row['time']}) for customer {customer_id}.\"
@mcp.tool
async def open_invoices(customer_id: int) -> list[dict]:
\"\"\"List a customer's invoices with their amounts and status.\"\"\"
return q(\"SELECT id, amount, status FROM invoices WHERE customer_id = ?\", (customer_id,))
@mcp.tool
async def issue_refund(invoice_id: int, ctx: Context) -> str | InputRequiredResult:
\"\"\"Refund an invoice. A human must approve the amount and give a reason.\"\"\"
inv = q(\"SELECT id, customer_id, amount, status FROM invoices WHERE id = ?\", (invoice_id,))
if not inv:
return f\"No invoice {invoice_id}.\"
inv = inv[0]
responses = ctx.input_responses
if responses is None:
params = ElicitRequestFormParams(
message=(
f\"Refund invoice {inv['id']} for customer {inv['customer_id']}? \"
f\"It is {inv['status']}, amount {inv['amount']:.2f}. \"
\"Confirm the amount to refund and give a reason.\"
),
requested_schema={
\"type\": \"object\",
\"properties\": {
\"amount\": {\"type\": \"number\", \"description\": \"Amount to refund\"},
\"reason\": {\"type\": \"string\", \"description\": \"Why this refund is being issued\"},
},
\"required\": [\"amount\", \"reason\"],
},
)
return InputRequiredResult(
result_type=\"input_required\",
input_requests={\"approve_refund\": ElicitRequest(method=\"elicitation/create\", params=params)},
)
answer = responses[\"approve_refund\"]
if answer.action != \"accept\":
return f\"Refund declined - invoice {inv['id']} unchanged.\"
w(\"UPDATE invoices SET status = 'refunded' WHERE id = ?\", (inv[\"id\"],))
return f\"Refunded {answer.content['amount']:.2f} on invoice {inv['id']}. Reason: {answer.content['reason']}\"
if __name__ == \"__main__\":
mcp.run(transport=\"http\", host=\"127.0.0.1\", port=8770, stateless_http=True)
The docstrings are not comments. They become the tool descriptions the model reads to choose,\nso write them as instructions to someone who can see nothing but that one line.
\nbook_slot refuses a taken slot in SQL rather than checking first ā WHERE customer_id IS NULL\nmeans two agents racing for the same slot can't both win.
cd ~/mcp-tools && ./.venv/bin/python office_tools.py
Leave it running. In a second session, ask what tools it has:
\ncurl -s -X POST http://127.0.0.1:8770/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}'
The first time we ran this, without stateless_http=True:
{\"jsonrpc\":\"2.0\",\"id\":null,\"error\":{\"code\":-32600,\"message\":\"Bad Request: Missing session ID\"}}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-session-error-7897aebee33998b591bd27b4606a6e6b.png\")
The 2026-07-28 revision removed sessions from the protocol ā no handshake, no session ID,\nnothing pinning a client to one instance. Every write-up of the change describes the client\nside accurately: there is nothing left to pin.
FastMCP 4.0.2 still defaults to the stateful transport. You opt in with\nstateless_http=True, and until you do, the first request against a brand-new server fails\nciting a concept the spec deleted five weeks ago.
run_http_async accepts both stateless_http and stateless; the docstring says the second\nis \"Alias for stateless_http for CLI consistency.\" One switch, two names, both defaulting to\nNone.
The startup line is where you check it:
\n![\"FastMCP](\"https://development-wec.wiline.com/docs/assets/images/mcp-server-start-9b984393ff73ca52793743fe31bbe9da.png\")
![\"FastMCP](\"https://development-wec.wiline.com/docs/assets/images/mcp-server-stateless-8cd7f1d5201b5dc974f4726c52b23f9f.png\")
With the flag, the same request returns the catalog ā no handshake, no session:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-tools-list-c1f510e50217be4d48c633e07dafa45d.png\")
issue_refund's schema contains only invoice_id ā the ctx: Context parameter is absent,\nbecause FastMCP injects it and the model can't set it. That's what makes a mandatory\nconfirmation mandatory.
import asyncio, os, sys
from langchain.agents import create_agent
from langchain.mcp import MCPAdapter
from langchain_openai import ChatOpenAI
model = ChatOpenAI(
base_url=\"http://127.0.0.1:4000/v1\",
api_key=os.environ[\"GATEWAY_KEY\"],
model=\"qwen-mid\",
temperature=0,
)
async def main():
async with MCPAdapter(\"http://127.0.0.1:8770/mcp\") as adapter:
tools = await adapter.list_tools()
print(\"discovered:\", [t.name for t in tools])
agent = create_agent(model, tools)
out = await agent.ainvoke({\"messages\": [{\"role\": \"user\", \"content\": sys.argv[1]}]})
print(\"\\n--- answer ---\")
print(out[\"messages\"][-1].content)
asyncio.run(main())
Every published MCP example points the model at Anthropic, OpenAI or Gemini. This one points at\nyour own LiteLLM gateway, which forwards to WEC\nInference ā the tools are local, the model is yours, nothing leaves the box.
\nKeep the key in a file rather than on the command line, where it would end up in your shell\nhistory and every screenshot:
\ncd ~/mcp-tools && printf 'GATEWAY_KEY=%s\\n' \"$KEY\" > .env && chmod 600 .env
cd ~/mcp-tools && set -a && source .env && set +a && ./.venv/bin/python agent.py \"Who is Maria and what invoices does she have?\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-agent-lookup-d1e719855fca0eb978d89518c2a03cac.png\")
That answer took two tool calls in sequence: find_customer(\"Maria\") to get her id, then\nopen_invoices(1) using it. Nothing routed that ā the model read five descriptions and worked\nout the order.
Now one that writes:
\ncd ~/mcp-tools && ./.venv/bin/python agent.py \"Book Maria into a free slot on 2026-09-04\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-agent-booking-cf43e46e8fbeb7003efd7d2ceba1ddc4.png\")
Three calls this time. And because a booking is a real change, check it somewhere the agent\ncan't reach:
\ncd ~/mcp-tools && ./.venv/bin/python -c \"
import sqlite3
for r in sqlite3.connect('office.db').execute('select s.id,s.day,s.time,coalesce(c.name,\\\"free\\\") from slots s left join customers c on c.id=s.customer_id order by s.day,s.time'): print(r)\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-verify-db-e3eda545965f9b86a42918a3fd4bc674.png\")
You can stand up an MCP server over a real datastore, point an agent at it, and have it chain\ndiscovered tools into a change you can verify in the database ā with the model on your own\ngateway rather than a vendor's.
Part 1's approval pause was something you put in the graph. This one comes from the tool, and\nthe agent has no way to skip it.
\nimport asyncio, json, os
from langchain.agents import create_agent
from langchain.mcp import MCPAdapter
from langchain_openai import ChatOpenAI
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command
model = ChatOpenAI(base_url=\"http://127.0.0.1:4000/v1\",
api_key=os.environ[\"GATEWAY_KEY\"], model=\"qwen-mid\", temperature=0)
def ask_human(req):
print(f\"\\n>>> {req['message']}\")
props = (req.get(\"requested_schema\") or {}).get(\"properties\", {})
content = {}
for name, spec in props.items():
raw = input(f\" {name} ({spec.get('type','string')}): \").strip()
content[name] = float(raw) if spec.get(\"type\") == \"number\" else raw
return {\"action\": \"accept\", \"content\": content}
async def main():
async with MCPAdapter(\"http://127.0.0.1:8770/mcp\") as adapter:
tools = await adapter.list_tools()
agent = create_agent(model, tools, checkpointer=InMemorySaver())
cfg = {\"configurable\": {\"thread_id\": \"refund-1\"}}
out = await agent.ainvoke(
{\"messages\": [{\"role\": \"user\", \"content\": \"Refund Maria's open invoice.\"}]}, cfg)
if \"__interrupt__\" not in out:
print(\"NO INTERRUPT:\", out[\"messages\"][-1].content); return
payload = out[\"__interrupt__\"][0].value
print(\"=== INTERRUPT ===\")
print(json.dumps(payload, indent=2))
responses = {r[\"key\"]: ask_human(r) for r in payload[\"requests\"]}
print(\"\\n=== RESUMING ===\")
out = await agent.ainvoke(Command(resume={\"responses\": responses}), cfg)
print(out[\"messages\"][-1].content)
asyncio.run(main())
The checkpointer and thread_id are not optional ā an interrupted run needs somewhere to\nwait. Same requirement as Part 1's approval pause.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-elicit-full-b00707c0653a77cd26dcf4221b184fa5.png\")
Read the order of events. The model resolved \"Maria's open invoice\" to invoice 2 by itself ā\nshe has two, and it picked the open one. Then it stopped. The amount and the reason were\ntyped by a human, and only then did the refund happen.
\nThe payload is {\"type\": \"mcp_elicitation\", \"tool_name\": \"issue_refund\", \"requests\": [...]}.\ntype is the discriminator, so a graph with several kinds of interrupt can tell which came\nfrom MCP. Each request carries a key, a message for a human, a mode of form or url,\nand for a form the requested_schema the answer must satisfy ā here two required fields, one\nof them a number. You resume with one answer per key.
And verify, again outside the agent:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-verify-refund-3f9c62f71c37c5e0339cf4d1a4da57df.png\")
You can make a tool refuse to finish without a human ā returning a typed form the model cannot\nfill in itself, pausing the run as a LangGraph interrupt, and resuming it with an answer a\nperson actually typed.
ctx.elicit is the old API, and its error lands in the model's contextEvery FastMCP elicitation example you will find calls await ctx.elicit(message, response_type)\ninside the tool. That is the handshake-era mechanism: it blocks mid-execution and speaks\nover the session's back-channel ā the back-channel the stateless transport doesn't have.
Our first attempt used it. The agent didn't pause ā but not because nothing was raised. FastMCP\nraised its era error exactly as documented, LangChain converted it into a ToolMessage with\nstatus=\"error\" (deliberately, \"instead of ending the run\"), and the model read that error and\nnarrated it as progress:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-elicit-old-api-ec02d789aa6970b70a130f83067c235b.png\")
No interrupt, exit code 0, and the invoice still open. Nothing was refunded and nothing is\nwaiting for a human ā but the user was told an approval is pending.
On the modern protocol a tool returns an InputRequiredResult describing what it needs, and\nexits. The client answers and retries the whole tool call with the answer attached. Each\nround is a complete request, which is why it survives load balancing and redeploys ā and why\nthe tool body must be written to be re-entered rather than resumed. Ours does that with\nif ctx.input_responses is None.
FastMCP's docs are explicit that ctx.elicit works only on connections ā¤ 2025-11-25, and that\na multi-era server should branch on ctx.request_context.protocol_version. They also promise the\nmismatch \"raises a clear era error rather than failing obscurely\" ā and it does. The error just\nlands in the model's context rather than yours.
LangChain's post reads the interrupt as paused[\"__interrupt__\"][0].value.requests[0] ā\nattribute access. MCPElicitationInterrupt is a TypedDict (see langchain/mcp/elicitation.py\nin 1.4.0), so at runtime it is a dict and .requests doesn't resolve. Use value[\"requests\"] ā\nas the next line of their own snippet does, reading question[\"key\"].
Every run starts by discovering tools, which is a round trip before the model sees anything.\nThe new spec makes that cacheable:
\nimport asyncio, time
from fastmcp import Client
from langchain.mcp import MCPAdapter
async def main():
client = Client(\"http://127.0.0.1:8770/mcp\", cache=True)
async with MCPAdapter(client) as adapter:
for i in (1, 2):
t = time.perf_counter()
tools = await adapter.list_tools(cache_mode=\"use\")
print(f\"discovery {i}: {len(tools)} tools in {(time.perf_counter()-t)*1000:.1f} ms\")
asyncio.run(main())
discovery 1: 5 tools in 14.6 ms
discovery 2: 5 tools in 12.5 ms
![\"Two](\"https://development-wec.wiline.com/docs/assets/images/mcp-cache-bbadf88f6b26a069ec15e3b3db635c02.png\")
No effect ā and that is correct behaviour, not a broken cache.
\ncache=True is necessary, not sufficientThe client cache \"respects the ttlMs and cacheScope hints the server attaches to each\nresponse\" and works only against modern-era servers that advertise them. Our server advertises\nnone, so both calls go to the network.
Note also that cache=True goes on the fastmcp.Client, not on MCPAdapter ā the\nadapter takes exactly one argument. And the cache belongs to the client, so one client per\ncaller keeps catalogs from crossing between tenants.
Over loopback there was nothing to save anyway: 12ā15 ms is the round trip. Caching is for remote\nservers with real latency and a TTL to honour.
\nResolutionImpossible installing mcp[cli]āThe cli extra was removed in mcp 2.x. Install \"langchain[mcp]\" instead; it pulls a\ncompatible mcp and fastmcp.
Bad Request: Missing session IDāYour server is running the stateful transport. Add stateless_http=True to mcp.run(...).
KeyError: 'GATEWAY_KEY'āsource .env sets shell variables, not environment variables ā the Python child never sees\nthem. Use set -a && source .env && set +a. And export does not cross SSH sessions, so a\nsecond terminal needs its own source.
address already in use after restarting the serverāThe previous instance is still holding the port ā a crashed restart leaves the old process\nalive, so the fix you just made appears not to work. pkill -f office_tools.py, then check\nwith ss -tlnp | grep 8770.
The tool is calling ctx.elicit rather than returning InputRequiredResult. On a stateless\nconnection there is no back-channel. The era error is raised, but it arrives as a failed\nToolMessage the model reads and talks past, so the run ends normally with nothing pending.\nCheck the tool messages, not just the final answer.
Done: an MCP server over a real datastore, an agent that discovers tools rather than being\nwired to them and chains them without routing, two changes you verified in the database rather\nthan taking the model's word for, and a refund that cannot happen without a human supplying the\nnumber.
\nNot done:
\n127.0.0.1 and anyone on the box can call every\ntool, including issue_refund. FastMCP supports bearer tokens and OAuth 2.1; we configured\nneither.book_slot will book for any customer_id it is\ngiven. There is no notion of who is asking ā the same gap\nPart 4 of the hardening series found\nbetween a control plane and a data plane.ClientGroup connects several at once and prefixes tool names by server; we\nused a single target.book_slot is doing\nmore work than it looks under any real concurrency.langchain.mcp is beta and warns on every import.Scope the tools per agent. Right now one agent holds all five, so nothing stops the model\nchoosing issue_refund when it was asked to book a slot ā the human pause is the only guard.\nPart 2's supervisor already routes between a scheduler and a billing agent; hand each of them\nonly its own slice of adapter.list_tools() and the scheduler cannot issue a refund because it\nhas never been told the tool exists. That is a stronger guarantee than asking a model nicely.
Then authentication, so a tool call carries an identity rather than trusting whoever reached\nthe port ā the same question Part 4 of the hardening series\nasked about the gateway's own API, one layer up.
\nlangchain-mcp-adapters2026-07-28 specificationPart 1 closed the ports nobody meant to\nopen. Part 2 stopped two containers\nfrom running as root. Both were about the machine. Neither touched the question a\nself-hosted AI stack answers worst: who is allowed to log in, and where is that\ndecided?
\nSame box as before. The Langfuse we've been hardening since Part 1 ā the one whose\nPostgres, ClickHouse and Redis Part 1 found correctly bound to localhost, and whose\nworker Part 2 dropped off root ā was deployed back in\nCatch what your tests miss. Two parts\nhave now secured the machine underneath it without once touching the application's own\nfront door. This is the first part that changes Langfuse itself.
\nRight now that decision is made in each app, separately. Langfuse has its own email-and-\npassword table. So does every other tool on the box. Each one is a place where an account\ncan outlive the person who owned it, where a password can be reused, and where \"remove\nthis person's access\" means remembering that the app exists at all.
\nThis post moves that decision to one place. That place is\nauthentik ā an open-source identity provider you run\nyourself, the self-hosted counterpart to Okta or Auth0. It holds the accounts, runs the\nlogin screen, and vouches for who someone is to any app that asks. Apps stop storing\npasswords and start asking authentik.
\nIt's the same job Keycloak does, and Keycloak is the better-known name. authentik earns\nthe pick here on setup cost: a compose file and a wizard against Keycloak's realms,\nclients and JVM tuning. For one box and one app, that difference is the whole decision.
\nWe deploy it, connect Langfuse to it over OIDC, and end with a login that goes through\nauthentik and back.
\n\nOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Docker 29.1.3, kernel 5.15.\nauthentik 2026.8.1 from ghcr.io/goauthentik/server, Postgres 16. Langfuse v3\n(langfuse/langfuse:3) ā the deployment carried over from Parts 1 and 2, reachable at\n10.80.4.212:3001. authentik published on host port 9100.
Addresses in this post are a private LAN. Substitute yours. Everything here is plain\nHTTP on a trusted network, which is fine for a first run and not fine for anything\nreachable from outside ā see What's next.
Every OIDC guide assumes you already know these five words. Getting them straight up\nfront is the difference between configuring this once and guessing at fields for an\nhour.
\nOP and RP. The OpenID Provider is the thing that knows who people are ā\nauthentik. The Relying Party is the app that wants to be told ā Langfuse. The RP\nnever sees a password. It receives a signed statement from the OP saying \"this is\nadmin@example.com, I checked.\"
Provider vs application. authentik splits what most tools merge. A provider is\nthe protocol endpoint ā the OIDC machinery, the client ID and secret, the redirect URI.\nAn application is the thing users see and the thing access rules attach to. They\npair one-to-one: one application, one provider, and the wizard in Step 3 creates both\ntogether.
\nThe redirect URI is where authentik is allowed to send the user back after a\nsuccessful login. It's matched exactly and it's the single most common source of\nfailure. Not a prefix, not a hostname ā the full URL, character for character.
\nScopes vs grant types. A scope is what information the app asks for\n(openid email profile). A grant type is the mechanism by which it asks\n(authorization_code). Different questions; the UI puts them near each other and it's\neasy to conflate them.
The part worth noticing: the code travels through the browser, the secret never does.\nThat exchange in the second-to-last step happens between the two containers directly,\nwhich is why the client secret is a server-side setting and why Langfuse must be able to\nreach authentik over the network ā not just your browser.
\n9100)localhost ā two containers need to reach each other and\nyour browser needs to reach bothauthentik ships a compose file and generates its own secrets. Create a directory and\npull both down:
\nmkdir -p ~/authentik && cd ~/authentik
curl -O https://goauthentik.io/docker-compose.yml
Now the environment file. authentik needs a Postgres password and a secret key, both\nrandom, plus the ports it will publish:
\ncd ~/authentik
{
echo \"PG_PASS=$(openssl rand -base64 36 | tr -d '\\n')\"
echo \"AUTHENTIK_SECRET_KEY=$(openssl rand -base64 60 | tr -d '\\n')\"
echo \"COMPOSE_PORT_HTTP=9100\"
echo \"COMPOSE_PORT_HTTPS=9543\"
} > .env
chmod 600 .env
AUTHENTIK_SECRET_KEY signs sessions and tokens. If you lose it, every existing session\nand every issued token becomes invalid ā so it belongs in your password manager, not\nonly in this file.
Bring it up. First boot runs database migrations and takes a minute or two:
\ncd ~/authentik && docker compose up -d
cd ~/authentik && docker compose ps
Three containers ā server, worker and postgresql ā all reporting (healthy).\nOlder authentik compose files also shipped a separate redis service; 2026.8.1 no\nlonger does, so three is correct and nothing is missing. server is the one that has to\nbe healthy before you continue.
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/authentik-compose-ps-2cdcd74dfeb2be5f3c8d9bb137b646db.png\")
authentik's first boot leaves the admin account unclaimed. The setup URL exists exactly\nonce; visiting it is how you take ownership.
\nOpen http://10.80.4.212:9100/if/flow/initial-setup/, and set the email and password\nfor the default admin (akadmin).
![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-initial-setup-e170a5fefdbc3d6a5b645819857b6cae.png\")
Use an address you control. This account can do everything, so give it a real password\nand store it somewhere durable.
\nUntil someone completes this form, anyone who can reach the port can become the\nadministrator of your identity provider. Do it immediately after the containers come up\nā not tomorrow.
You land on the Application Dashboard ā the user-facing view, not the admin one, and empty\nbecause nothing is configured yet.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-app-dashboard-empty-fb4aaff0a94a00ac91abe85b9541079d.png\")
Create a new application here goes to the same place as the admin interface does; the\nAdmin interface button top-right is how you reach everything else.
Applications ā Applications ā Create. authentik walks you through five steps, and\nthe step list on the left is worth reading before you touch anything:
\nApplication ā Choose a Provider ā Configure Provider ā Configure Bindings ā Review and\nSubmit
\nThat's the vocabulary from earlier, in order. The wizard creates the application and its\nprovider in one pass and pairs them for you. (Applications ā Providers ā Create builds\na provider on its own, if you ever need one before the app that uses it.)
Langfuse. The label users see on the dashboard.langfuse. Load-bearing: it becomes part of the issuer URL, so changing it\nlater changes the URL you have to reconfigure Langfuse with.![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-wizard-application-62b3555c7b9b05dedfdd535153d2c084.png\")
Eight provider types. Take OAuth2/OpenID Provider ā \"OAuth2 Provider for generic\nOAuth and OpenID Connect Applications.\"
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-wizard-provider-type-62af4193167bfaa5359808b786af8d41.png\")
Worth knowing what you're not picking, because two of these solve a different problem:\nProxy Provider puts authentik in front of an app that has no SSO support of its own,\nand LDAP Provider exposes authentik to things that only speak LDAP. Langfuse speaks\nOIDC natively, so it gets a real OIDC provider.
\nThe step that matters, and the one that will cost you time if you rush it.
\nAuthorization flow ā default-provider-authorization-explicit-consent. \"Explicit\"\nshows the user a consent screen listing what Langfuse asked for. The implicit variant\nskips it. Pick explicit for a first build: that screen is a live readout of your scope\nconfiguration, which makes a misconfiguration visible instead of silent.
Client type ā Confidential (scrolled above the redirect fields). Langfuse is a\nserver and can keep a secret, so it gets one. Public clients are for browsers and\nmobile apps that can't.
\nRedirect URIs/Origins (RegEx) ā the field with two dropdowns and a value. Set the\nmode to Strict, the purpose to Authorization, and the URI to Langfuse's\nNextAuth callback:
\nhttp://10.80.4.212:3001/api/auth/callback/custom
The trailing custom is not a placeholder ā it's the provider ID NextAuth assigns to\nits generic OIDC provider. Use your own host and port, but that path is fixed.
Signing Key ā leave authentik Self-signed Certificate. This signs the ID token;\nLangfuse fetches the matching public key from the jwks_uri in Step 5 and verifies\nagainst it. Self-signed is correct here ā the RP trusts this key because discovery\nhanded it over, not because a CA vouched for it.
Logout URI and token validities ā leave alone.
\nThe provider once it's saved ā explicit-consent authorization flow, Confidential client\ntype, and Authorization Code as the grant:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-wizard-configure-provider-11b9bd02c65bf52f23d2afe34f3a8be9.png\")
The mode dropdown offers Regex as well, and authentik's own help text points out you\ncan set it to .* to allow any redirect URI. Don't. An open redirect URI means anyone\nwho can start a login can have the authorization code delivered to a host they control.\nStrict, with one exact URL, is the whole point of the field.
Further down, Scopes comes pre-populated with four mappings: email, openid,\nprofile and offline_access. Leave them. openid is mandatory and is what marks this\nas OIDC rather than bare OAuth2. email is the claim Langfuse keys the account on.\nprofile carries the display name. offline_access is the one the wizard adds without\nasking ā it permits a refresh token, so a session can be renewed without sending the user\nback through the login flow.
No bound policies. ā and we are leaving it that way.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-wizard-bindings-b5cb6f9b81084f2eb498d0b4cdeae6cd.png\")
Read the wizard's own description: \"These policies control which users can access this\napplication.\" With nothing bound, the answer is everyone who can authenticate. On a\nbox where you're the only account, that's the same thing as \"just me\" ā which is why\nit's tolerable here and why the wizard lets you skip it.
It is not access control. Every account you ever create in this authentik gets into\nLangfuse by default, and nothing will warn you. Groups and policy bindings are how that\ngets fixed, and they're the subject of the next post ā where the thing being protected\nis a gateway and the answer matters a great deal more.
Check the slug and the redirect URI once more, then submit.
\nRather than assembling URLs by hand, ask authentik. Every OIDC provider publishes a\ndiscovery document, and authentik's lives under the application slug:
\ncurl -s http://10.80.4.212:9100/application/o/langfuse/.well-known/openid-configuration \\
| jq '{issuer, authorization_endpoint, token_endpoint, userinfo_endpoint, jwks_uri, grant_types_supported}'
{
\"issuer\": \"http://10.80.4.212:9100/application/o/langfuse/\",
\"authorization_endpoint\": \"http://10.80.4.212:9100/application/o/authorize/\",
\"token_endpoint\": \"http://10.80.4.212:9100/application/o/token/\",
\"userinfo_endpoint\": \"http://10.80.4.212:9100/application/o/userinfo/\",
\"jwks_uri\": \"http://10.80.4.212:9100/application/o/langfuse/jwks/\",
\"grant_types_supported\": [
\"authorization_code\",
\"refresh_token\",
\"implicit\",
\"client_credentials\",
\"password\",
\"urn:ietf:params:oauth:grant-type:device_code\"
]
}
Two things to take from this. The issuer is the only URL Langfuse needs ā it will\nfetch this same document and discover the rest. And grant_types_supported lists what\nthe protocol offers, not what your provider will accept; we're using\nauthorization_code and the presence of password in that list is not an invitation.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-discovery-json-959875837069e04a0b42b7cc81ad0b75.png\")
Now the client credentials. They're in the provider's detail page in the UI, but\ndocker compose exec is faster and copy-pastes cleanly:
cd ~/authentik && docker compose exec server ak shell -c \"
from authentik.core.models import Application
p = Application.objects.get(slug='langfuse').get_provider()
print('PROVIDER :', p.name)
print('CLIENT_ID :', p.client_id)
print('CLIENT_SECRET:', p.client_secret)
\"
Note that this looks the provider up through the application slug rather than by\nprovider name. The wizard named your provider Provider for Langfuse, not langfuse ā\nthe slug is the thing you chose and the thing that stays predictable. get_provider()\nalso matters: reaching for application.provider hands back the base class with\nclient_id empty, which looks alarmingly like a broken provider and isn't.
Keep that output out of your shell history and out of screenshots.
\nThe provider's detail page carries the same values, if you'd rather read them there ā\nclient type, client ID, the strict redirect URI, and every discovery URL in one place.\nThe client secret is the one thing it doesn't print:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-provider-overview-5b451791d184236f85921a5a394f956e.png\")
Langfuse reads generic OIDC settings from AUTH_CUSTOM_* variables. Add them to\n~/langfuse/.env:
AUTH_CUSTOM_NAME=authentik
AUTH_CUSTOM_ISSUER=http://10.80.4.212:9100/application/o/langfuse
AUTH_CUSTOM_CLIENT_ID=<the client id from Step 4>
AUTH_CUSTOM_CLIENT_SECRET=<the client secret from Step 4>
AUTH_CUSTOM_SCOPE=openid email profile
The issuer here has no trailing slash while the discovery document reported one. Both\nwork ā Langfuse normalizes it.
\nAlso confirm NEXTAUTH_URL matches the address you actually browse to:
grep NEXTAUTH_URL ~/langfuse/.env
NEXTAUTH_URL=http://10.80.4.212:3001
If this says localhost while your redirect URI says 10.80.4.212, the callback will\nbe built with the wrong hostname and authentik will reject it.
Now the part that will waste your afternoon if you skip it. Writing those variables\ninto .env is not enough. Langfuse's shipped compose file lists the variables it passes\ninto the langfuse-web container explicitly, and AUTH_CUSTOM_* isn't among them ā\nthey land in .env, get read by Compose for interpolation, and never reach the process.\nLangfuse starts fine and shows no SSO button, with nothing in the logs to say why.
Pass them through with an override file, ~/langfuse/docker-compose.override.yml:
services:
langfuse-web:
environment:
AUTH_CUSTOM_NAME: ${AUTH_CUSTOM_NAME}
AUTH_CUSTOM_ISSUER: ${AUTH_CUSTOM_ISSUER}
AUTH_CUSTOM_CLIENT_ID: ${AUTH_CUSTOM_CLIENT_ID}
AUTH_CUSTOM_CLIENT_SECRET: ${AUTH_CUSTOM_CLIENT_SECRET}
AUTH_CUSTOM_SCOPE: ${AUTH_CUSTOM_SCOPE}
AUTH_CUSTOM_ALLOW_ACCOUNT_LINKING: \"true\"
Compose merges docker-compose.override.yml automatically, so the upstream file stays\nuntouched and survives upgrades. That last variable is the fix for a failure you'd\notherwise hit in the next step ā the troubleshooting section explains what it does and\nwhat it costs.
Recreate the web container and verify the variables arrived:
\ncd ~/langfuse && docker compose up -d langfuse-web
cd ~/langfuse && docker compose exec langfuse-web env | grep -c AUTH_CUSTOM
Six. If it's zero, the override isn't being picked up ā check you're running compose\nfrom ~/langfuse and that the filename is exactly docker-compose.override.yml.
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/langfuse-env-count-c57cd6125a9c2dbc129ef850beee6378.png\")
Open http://10.80.4.212:3001/auth/sign-in, in a private window so you're not looking at\nan existing session. There's a new button below the divider under the password form.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/langfuse-sso-button-40985ca287c0686b8a2c65f8b7998d3d.png\")
The button is labelled authentik because that is your AUTH_CUSTOM_NAME value, rendered\nverbatim. Set it to Company SSO and that's what the button says ā it's a display string\nand nothing else depends on it.
Two other things in that screenshot are worth naming rather than glossing over. The\nemail-and-password form is still there, and so is Sign up ā SSO has been added, not\nsubstituted. And the address bar says Not Secure, correctly: this is plain HTTP.
\nClick the button and you're handed to authentik, which tells you where you came from:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-login-langfuse-f20d6f5f65651cb88e1db77bdd2f7c07.png\")
Log in, and because we chose the explicit-consent flow in Step 3.3, authentik shows what\nLangfuse asked for before it hands anything over:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/authentik-consent-1db931f804b8b9ddaa8562e01b588adf.png\")
It reads You're about to sign into Langfuse, identifies you as akadmin, and lists two\npermissions ā Email address and General Profile Information. Those are the email\nand profile scopes in human-readable form. openid doesn't get a line of its own: it\ncarries no personal data to consent to, it's the flag that makes this OIDC rather than\nplain OAuth2. So a three-scope configuration produces a two-item consent screen, which is\ncorrect and not a sign that something dropped.
Continue, and you land in Langfuse, signed in, as the identity authentik vouched for.
\nThat's the loop closed. Langfuse never saw a password.
\nOAuthAccountNotLinkedāThe first sign-in attempt bounced back to the Langfuse login page with\n?error=OAuthAccountNotLinked in the URL and nothing more.
The cause: an email-and-password Langfuse account already existed with the same address\nauthentik was now presenting. NextAuth's default is to refuse the merge. That default is\nprotective ā if an identity provider can assert any email and the app auto-links on\nemail alone, then whoever controls the provider can take over an existing account by\nclaiming its address.
\nAUTH_CUSTOM_ALLOW_ACCOUNT_LINKING: \"true\" tells Langfuse to link them anyway. It is\nsafe here because you own the only provider and you control who can create accounts\nin it. It would not be safe pointed at a provider where anyone can self-register an\narbitrary email.
The alternative, if you'd rather not enable it: delete the pre-existing local account\nand let SSO create a fresh one.
\nredirect_uri mismatch, or authentik refusing to redirectāRead the failing URL in the address bar and compare it to the provider's redirect URI\ncharacter by character. In this run the mismatches worth naming were the trailing\n/api/auth/callback/custom path (any other suffix fails), localhost versus the LAN IP,\nand a stray trailing slash. Strict matching means strict.
That's the compose-environment trap from Step 5. Check the variables reached the\ncontainer:
\ncd ~/langfuse && docker compose exec langfuse-web env | grep AUTH_CUSTOM_ISSUER
Empty output means Langfuse is running without the config it never knew it was missing.
\nYour browser reaching authentik is not proof that Langfuse can. The token exchange is\ncontainer-to-container:
\nThe langfuse-web image ships no curl, so use the Node runtime that's already in\nthere ā fetch is built in:
cd ~/langfuse && docker compose exec langfuse-web node -e \\
\"fetch('http://10.80.4.212:9100/application/o/langfuse/.well-known/openid-configuration').then(r=>console.log(r.status)).catch(e=>console.log('FAIL',e.message))\"
200 is what you want. FAIL with a connection error means the container can't reach\nauthentik at all, which is the actual thing being tested. This is also why the issuer is a LAN IP rather than localhost\nā inside the Langfuse container, localhost is the Langfuse container.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/langfuse-discovery-200-f17cc0564e966cf97ba8d8a4cc9e7c9f.png\")
You can run your own identity provider and put an app behind it over OIDC ā reading the\nissuer, client ID and secret out of authentik rather than assembling URLs by hand, and\nrecognising the two failures that actually stop you: a compose file that never passes the\nvariables through, and OAuthAccountNotLinked.
Done: one identity provider, one place where accounts live, one place to revoke them.\nLangfuse no longer stores a password for you. And you now have a provider you can point\nthe next app at in about five minutes.
\nNot done, and worth being clear about:
\nakadmin, you lose the provider that\nfronts everything pointed at it.Part 4 puts identity in front of something where these gaps stop being tolerable: the\nLiteLLM gateway from the gateway series.\nThree things this post deliberately left alone:
\n127.0.0.1:4000, and getting it\nproperly reachable means a reverse proxy, a hostname and a certificate.The provider-and-application steps will be a short recap with a link back here. The rest\nis new.
\nAUTH_CUSTOM_* variablesOAuthAccountNotLinked exists+Part 4 built a callback that masks\nPII in what the gateway logs without touching what the caller receives. It also noted\nthat the first working version used a blocking HTTP client inside an async def, and\nthat this cost 200-350ms across four sequential requests.
That post ended with an admission: four requests in a row is the wrong test. A blocked\nevent loop barely shows when nothing else is waiting. The damage should appear under\nconcurrency, and we had not measured it.
\nThis is that measurement, and the result is a shape rather than a number. With the\nblocking client, some batches take many times longer than they should. With the async\nclient, none do. The medians barely differ, which is exactly why this ships unnoticed.
\nTwo things turned up that were not in the plan. The gateway does not merely slow down ā\nit drops requests. And when it does, the masking call is the thing that timed out,\nwhile the caller still gets 200 OK. That means the guarantee Part 4 was built on\nquietly stops holding under load.
httpx in your Python environmentOne WEC Instance: 8 vCPU AMD EPYC 7601, 15 GB RAM, Docker 29.1.3, kernel 5.15. LiteLLM\n1.96.2 from ghcr.io/berriai/litellm:main-stable. Presidio analyzer and anonymizer as\nlocal containers on the same host. Model Qwen2.5-3B-Instruct over the WEC Inference\nAPI. Five repetitions per cell.
Latency numbers do not transfer between machines. Treat every second in this post as one\nobservation from that setup, not a figure to match. What should reproduce is the\ndifference between the two versions.
Everything here depends on one number, and it is not the number the documentation gives.
\nThe LiteLLM proxy CLI has a --num_workers flag. The\npublished reference says its default is\n\"Number of logical CPUs in the system, or 4 if that cannot be determined.\" On an\neight-core box that would mean eight worker processes, eight event loops, and a blocking\ncall stalling one eighth of your traffic.
Ask the host what is actually running ā from outside the container, so nothing inside it\ncan misreport:
\ndocker top llm-gateway
UID PID PPID C STIME TTY TIME CMD
root 2477458 2477435 3 16:43 ? 00:01:19 /app/.venv/bin/python3 /app/.venv/bin/litellm --config /app/config.yaml --port 4000
root 2481091 2477458 0 16:44 ? 00:00:05 /opt/prisma/binaries/node_modules/prisma/query-engine-debian-openssl-3.0.x -p 53053
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/gw5-one-process-a28a66c6eafcb70fa88a41c347ddd22d.png\")
Figure 1. One python process runs the whole proxy. The second entry is Prisma, the\ndatabase client, which does not serve requests.
\nIf --num_workers were greater than one you would see several python processes with the\nfirst one as their parent. Confirm it in the startup log:
docker logs llm-gateway 2>&1 | grep -E \"Started server process|Uvicorn running\" | tail -2
INFO: Started server process [1]
INFO: Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw5-startup-log-df1d1c66b00b54c6d3cca0e6787ae4eb.png\")
Figure 2. One server process. This log accumulates across restarts, so tail -2\nkeeps you looking at the current one.
Now ask the tool what its default is:
\ndocker exec llm-gateway litellm --help | grep -A5 \"^ --num_workers\"
--num_workers INTEGER Number of worker processes for uvicorn /
gunicorn, or Granian worker processes
(--workers). Default is 1 (from
DEFAULT_NUM_WORKERS_LITELLM_PROXY). With
--run_granian, use --granian_threads for
runtime threads per worker.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw5-worker-default-a862f11836711c637727b9c24b5d3b0a.png\")
Figure 3. The tool's own help text: \"Default is 1.\"
\nThat is a help string. Since we are about to contradict the published documentation, take\nit from the source too ā adjust the python version in the path to match your image:
\ndocker exec llm-gateway grep -rn 'DEFAULT_NUM_WORKERS_LITELLM_PROXY' \\
/app/.venv/lib/python3.13/site-packages/litellm/constants.py
15:DEFAULT_NUM_WORKERS_LITELLM_PROXY = int(os.getenv(\"DEFAULT_NUM_WORKERS_LITELLM_PROXY\", 1))
The CLI reference states the --num_workers\ndefault as \"Number of logical CPUs in the system, or 4 if that cannot be determined.\"\nThree sources say otherwise: the running container, the installed 1.96.2 source, and\ncurrent upstream main, where both the constant and the help string still say 1.
It is not a version skew, and it is not something in this deployment. Check yours:
docker exec llm-gateway env | grep -i worker
Empty output means you are on the default, which is one worker.
So: one process, one event loop, shared by every request in flight. That is the condition\nthat turns a blocking call from a tax into an outage.
\nFire N requests at once, time each, report the wall clock for the batch. The message\ncarries PII so the masking callback has real work to do. A failed request is counted\nrather than allowed to abandon the run ā you will need that.
\n\"\"\"Fire N chat completions at the gateway at once and report how long they take.
The message carries PII so the Presidio masking callback has real work to do.
A failed request is counted and reported, not allowed to abandon the batch.
\"\"\"
import asyncio
import os
import statistics
import sys
import time
import httpx
GATEWAY = \"http://127.0.0.1:4000/v1/chat/completions\"
KEY = os.environ[\"LITELLM_MASTER_KEY\"]
MODEL = \"qwen-small\"
PROMPT = (
\"Priya Raghunathan called from 415-555-0134 about her invoice. \"
\"Reply with exactly: ok\"
)
async def one(client, i):
start = time.perf_counter()
try:
r = await client.post(
GATEWAY,
headers={\"Authorization\": f\"Bearer {KEY}\"},
json={
\"model\": MODEL,
\"messages\": [{\"role\": \"user\", \"content\": PROMPT}],
\"max_tokens\": 5,
},
)
except Exception as exc:
return i, None, None, type(exc).__name__
return i, r.status_code, time.perf_counter() - start, None
async def main():
n = int(sys.argv[1])
async with httpx.AsyncClient(timeout=120.0) as client:
wall = time.perf_counter()
results = await asyncio.gather(*(one(client, i) for i in range(n)))
wall = time.perf_counter() - wall
times = sorted(e for _, _, e, _ in results if e is not None)
errors = [err for _, _, _, err in results if err]
print(f\"concurrency {n} ok {len(times)} failed {len(errors)}\")
if errors:
print(f\" errors {', '.join(sorted(set(errors)))}\")
print(f\" wall {wall:.2f}s\")
if times:
print(f\" fastest {times[0]:.2f}s\")
print(f\" median {statistics.median(times):.2f}s\")
print(f\" slowest {times[-1]:.2f}s\")
asyncio.run(main())
cd ~/llm-gateway && set -a && . ./.env && set +a
~/.venv/bin/python loadtest.py 8
concurrency 8 ok 8 failed 0
wall 1.93s
fastest 1.60s
median 1.91s
slowest 1.93s
![\"Eight](\"https://development-wec.wiline.com/docs/assets/images/gw5-first-load-be7eeba783710fd2f310b7add1c21fdc.png\")
Figure 4. Eight at once, none failed. On its own this number means nothing yet.
\nHere is the trap. The gateway calls a model over the network. That model has its own\nlatency, its own load, and its own bad afternoons. When a batch takes twelve seconds you\ncannot tell whether your gateway stalled or the upstream was busy ā and if you guess, you\nwill publish nonsense.
\nSo measure the upstream directly, with the gateway out of the path. Copy the file and\nchange the three constants at the top:
\ncd ~/llm-gateway
cp loadtest.py loadtest_direct.py
GATEWAY = \"https://inference.wiline.com/v1/chat/completions\"
KEY = os.environ[\"WEC_API_KEY\"]
MODEL = \"Qwen2.5-3B-Instruct\"
~/.venv/bin/python loadtest_direct.py 8
concurrency 8 ok 8 failed 0
wall 1.15s
fastest 1.05s
median 1.10s
slowest 1.15s
![\"Eight](\"https://development-wec.wiline.com/docs/assets/images/gw5-control-6c9353892f98ded0047bab740eae67a0.png\")
Figure 5. The same eight requests with the gateway removed. This is the baseline\nevery gateway number gets read against.
\nNow run the two interleaved, not one after the other, so a slow minute upstream\ncannot land entirely on one version:
\n#!/bin/bash
# Interleave gateway runs with direct-to-upstream control runs so an upstream
# slowdown cannot be mistaken for a gateway effect.
set -a; . ./.env; set +a
REPS=5
# A restart takes longer than it looks. Firing requests at a port that is not
# answering yet produces instant failures that have nothing to do with the callback.
printf 'waiting for the gateway'
for _ in $(seq 1 60); do
curl -sf http://127.0.0.1:4000/health/liveliness >/dev/null 2>&1 && break
printf '.'; sleep 2
done
curl -sf http://127.0.0.1:4000/health/liveliness >/dev/null 2>&1 || { echo \" never came up ā aborting\"; exit 1; }
echo \" up\"
# Do not trust a label passed on the command line ā ask the gateway which callback
# it actually loaded. A mislabelled run is worse than no run.
LABEL=$(docker logs llm-gateway 2>&1 | grep '\\[scrubber\\]' | tail -1 | sed 's/.*loaded: //')
if [ -z \"$LABEL\" ]; then echo \"cannot determine loaded callback ā aborting\"; exit 1; fi
echo \"callback in use: $LABEL\"
summarise() {
awk '/^concurrency/{failed=$6} /wall/{wall=$2} END{printf \"%s\", wall; if (failed+0 > 0) printf \"(%d failed)\", failed}'
}
# The first request after a restart pays for imports and connection pools, which
# has nothing to do with the callback. Warm both paths before recording anything.
~/.venv/bin/python loadtest.py 4 >/dev/null 2>&1
~/.venv/bin/python loadtest_direct.py 4 >/dev/null 2>&1
for n in 8 16; do
for i in $(seq 1 $REPS); do
g=$(~/.venv/bin/python loadtest.py $n | summarise)
d=$(~/.venv/bin/python loadtest_direct.py $n | summarise)
echo \"$LABEL N=$n rep=$i gateway=$g direct=$d\"
done
done
Three details in there are not decoration, and each one cost a wasted run to learn:
\nIt waits for the gateway. A docker compose restart can take longer than ten\nseconds, and firing at a port that is not listening yet produces instant failures ā\ngateway=0.03s(8 failed) ā that look like a catastrophic result and mean nothing.
It refuses to take a label from you. It reads the loaded callback out of the\ngateway's log. Pass blocking on the command line while the async callback is loaded and\nyou will measure the same code twice, see no difference, and conclude there is nothing\nhere. That happened twice while writing this.
It throws away a warm-up pass. The first batch after a restart pays for imports and\nconnection pools, and comes in three to six times slower than the next one regardless of\nwhich callback is loaded.
\nYou are about to compare two versions of one file, and you need certainty about which is\nlive. LiteLLM's startup log lists success and failure callbacks but not\nlitellm_settings.callbacks, and there is no endpoint that reports them ā\n/get/config/callbacks answers 200 with {\"detail\":\"Not Found\"}.
So have each version say its own name at import, where it costs nothing per request:
\n# last line of scrubber.py
print(\"[scrubber] loaded: async httpx.AsyncClient\", flush=True)
# last line of scrubber_blocking.py
print(\"[scrubber] loaded: blocking httpx.Client\", flush=True)
Mount both files, and switch with one line of config:
\n volumes:
- ./config.yaml:/app/config.yaml:ro
- ./scrubber.py:/app/scrubber.py:ro
- ./scrubber_blocking.py:/app/scrubber_blocking.py:ro
- ./recognizers.json:/app/recognizers.json:ro
litellm_settings:
callbacks: [\"scrubber.instance\"] # or scrubber_blocking.instance
sed -i 's/scrubber.instance/scrubber_blocking.instance/' config.yaml
docker compose restart litellm
The experiment script prints the loaded callback as its first line, so every run carries\nits own proof of what it measured.
\n./run_experiment.sh
With the async client:
\ncallback in use: async httpx.AsyncClient
async httpx.AsyncClient N=8 rep=1 gateway=0.98s direct=1.09s
async httpx.AsyncClient N=8 rep=2 gateway=0.86s direct=0.80s
async httpx.AsyncClient N=8 rep=3 gateway=0.92s direct=0.96s
async httpx.AsyncClient N=8 rep=4 gateway=0.78s direct=0.85s
async httpx.AsyncClient N=8 rep=5 gateway=0.90s direct=0.82s
async httpx.AsyncClient N=16 rep=1 gateway=1.61s direct=1.49s
async httpx.AsyncClient N=16 rep=2 gateway=1.65s direct=1.26s
async httpx.AsyncClient N=16 rep=3 gateway=1.28s direct=1.35s
async httpx.AsyncClient N=16 rep=4 gateway=1.33s direct=1.42s
async httpx.AsyncClient N=16 rep=5 gateway=1.77s direct=1.56s
![\"Ten](\"https://development-wec.wiline.com/docs/assets/images/gw5-async-runs-44f396a0682d60c0748426e50abc69cf.png\")
Figure 6. The async client. Every gateway number sits beside its control, at both\nconcurrency levels. Nothing stands out because nothing happened.
\nThen swap the callback and run the same thing:
\nwaiting for the gateway up
callback in use: blocking httpx.Client
blocking httpx.Client N=8 rep=1 gateway=1.20s direct=0.93s
blocking httpx.Client N=8 rep=2 gateway=0.96s direct=0.83s
blocking httpx.Client N=8 rep=3 gateway=0.92s direct=0.86s
blocking httpx.Client N=8 rep=4 gateway=0.73s direct=0.90s
blocking httpx.Client N=8 rep=5 gateway=1.70s direct=0.90s
blocking httpx.Client N=16 rep=1 gateway=70.18s direct=2.00s
blocking httpx.Client N=16 rep=2 gateway=1.75s direct=1.70s
blocking httpx.Client N=16 rep=3 gateway=2.33s direct=1.78s
blocking httpx.Client N=16 rep=4 gateway=2.88s direct=1.49s
blocking httpx.Client N=16 rep=5 gateway=5.14s direct=1.19s
![\"Ten](\"https://development-wec.wiline.com/docs/assets/images/gw5-blocking-runs-df9535bdd0ed417cb65599e60ca5e742.png\")
Figure 7. The blocking client. Same script, same load, one word different in the\ncode. The control column holds at 1.19-2.00s throughout.
\n| Version | N | Gateway median | Gateway worst | Control median |
|---|---|---|---|---|
async AsyncClient | 8 | 0.90s | 0.98s | 0.85s |
async AsyncClient | 16 | 1.61s | 1.77s | 1.42s |
blocking Client | 8 | 0.96s | 1.70s | 0.90s |
blocking Client | 16 | 2.88s | 70.18s * | 1.70s |
Figure 8. Every batch from Figures 6 and 7 on a log axis. The control stays inside a\nnarrow band in all four groups. The async runs stay with it. Two blocking runs leave it.
\nThe 70.18s batch overlapped an unrelated load test running against the same gateway, so\npart of it is not attributable to the callback. It is reported because it happened, not\nbecause it is clean. Read the 5.14s batch as the representative stall ā still more than\nfour times its own control of 1.19s.
The effect itself reproduced across four separate runs on this host, with N=16 stalls of\n11.22s, 12.44s, 21.22s, 31.14s and 5.14s. Across every clean async run: none.
Read the async rows first. At N=8 the gateway's median is 0.90s and the control's is\n0.85s. The callback, Presidio round trips and all, disappears into the model's own\nlatency.
\nNow the blocking row at N=8. Median 0.96s against a 0.90s control. That is 0.06s. If you\nmeasured medians and shipped, you would call it fine ā and the worst batch in the same\nfive took 1.70s while its own control took 0.90s.
\nAt N=16 it stops hiding. The median goes to 2.88s against 1.70s, and the tail runs off\nthe chart.
\nYou can tell whether a slow gateway is your own code or the model it calls ā run the same\nload against both, interleaved, and read the tail instead of the median.
Before failure counting was added to the load script, a blocking run died outright:
\nhttpx.RemoteProtocolError: Server disconnected without sending a response.
Two of ten batches in that run lost requests that way. Zero async batches ever did. The\ngateway was not slow for those callers. It hung up on them.
\nAsk the gateway what it thinks happened:
\ndocker logs llm-gateway 2>&1 | grep -c \"httpcore/_sync\"
docker logs llm-gateway 2>&1 | grep -A1 \"httpx.ReadTimeout\" | tail -4
77
INFO: 172.22.0.1:47566 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK
--
httpx.ReadTimeout: timed out
INFO: 172.22.0.1:52870 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK
![\"Seventy-seven](\"https://development-wec.wiline.com/docs/assets/images/gw5-sync-timeout-d454e62c9e5512f71923e0ec2c0efbec.png\")
Figure 9. httpcore/_sync appears 77 times. The async client would produce\n_async. And the timeout sits between two 200 OK lines.
That _sync is the fingerprint. It proves the failing call is the blocking Presidio call\nand not something else in the stack. The full error names the caller:
LiteLLM:ERROR: logging_worker.py:103 - LoggingWorker error: timed out
File \".../httpcore/_sync/connection_pool.py\", line 236, in handle_request
httpcore.ReadTimeout: timed out
Now look at what surrounds it. 200 OK. The requests succeeded. The masking is what\nfailed.
That is the part worth stopping on, because Part 4's whole promise was that PII reaches\nyour traces already masked. Under load, with the blocking client, the masking call times\nout against its own ten-second limit while the caller receives a normal success. Nothing\nin the response tells you the guarantee lapsed. You would have to be reading the\ngateway's stderr to know.
\nhttpx.Client inside an async def does not yield. When the callback calls Presidio, the\nthread running the event loop sits in a socket read until Presidio answers, and during\nthat time the loop services nothing ā not another request's model call, not a response\ncoming back. It is one loop, as you confirmed at the start.
Each request triggers several of these calls: the messages, the copy in the standard\nlogging object, and the response. So under concurrency the requests do not overlap. They\nqueue, and each one's wait is every earlier one's Presidio time added together. That is\nwhy the effect is not a constant tax ā it depends on how many requests happen to arrive\nwhile the loop is held, which is also why the numbers are spiky rather than uniformly\nworse.
\nPast a certain queue depth the waits exceed the scrubber's own timeout=10.0 and the\nmasking gives up, while client connections waiting on a frozen loop get dropped. The\nmedian hides all of it because most batches get lucky. The tail is the truth.
The fix is the one Part 4 landed on, and this is the evidence for it: httpx.AsyncClient\nwith await, which yields the loop while Presidio works.
diff scrubber.py scrubber_blocking.py
26,27c26,27
< async with httpx.AsyncClient(timeout=10.0) as client:
< r = await client.post(f\"{ANALYZER}/analyze\", json={\"text\": text, \"language\": \"en\"})
---
> with httpx.Client(timeout=10.0) as client:
> r = client.post(f\"{ANALYZER}/analyze\", json={\"text\": text, \"language\": \"en\"})
Two words and an await. That is the whole difference between Figure 6 and Figure 7.
More workers is not the fix, and the documentation explains why better than we could. On\n--timeout_worker_healthcheck, describing --num_workers > 1:
\n\n\"the supervisor process pings each worker; a worker that does not respond within this\nwindow (for example because its event loop is blocked by synchronous work) is killed\nwith SIGKILL and replaced.\"
\n
At one worker there is no supervisor, so a blocked loop stalls. At several, a blocked\nworker gets killed and everything in flight on it dies instead of waiting. Neither is a\nfix for synchronous work on an event loop. Fix the call.
\nFive reps per cell is enough to show that a stall exists next to a steady control. It is\nnot enough to characterise the distribution ā how often, how bad, or how it scales past\nN=16. If you run this in production, run it longer and look at percentiles.
\nWe did not test --num_workers > 1, streaming responses, or a slow or remote Presidio\nrather than a healthy local container.
The obvious worry, when the masking call times out on a request that returned 200 OK,\nis that the unmasked payload lands in the trace anyway. It does not.
Sustained load ā six rounds of twenty-four concurrent requests, each carrying a name and\na phone number ā produced 77 masking timeouts in the gateway log. Of the 144\nrequests, 62 traces reached Langfuse and 82 never arrived at all. Every one of the 62\nwas properly masked. None carried a raw name.
\nSo the failure is not a privacy failure. It is an observability failure, and a quiet one:\nunder sustained load more than half the traffic simply is not recorded, while every\ncaller gets a normal success. If you are reading trace volume as a proxy for traffic, or\ncounting on traces for an audit trail, that gap is the thing to watch ā and it is\ninvisible from the response side.
\nBuilding this test, requests were tagged with a short marker so each trace could be found\nā [TAG-07] Priya Raghunathan called from ā¦. Nine traces then came back with the phone\nmasked and the name in the clear, which reads exactly like a load-induced leak.
It was not. Sequentially, with no load at all, that sentence sent straight to Presidio\nreturns only PHONE_NUMBER; the same sentence without the bracketed prefix returns\nPERSON and PHONE_NUMBER both. The marker suppressed name detection, and the callback\nfaithfully masked everything Presidio reported.
Check what your analyzer detects in your exact strings before concluding anything about\nmasking under load. An instrument that changes the thing it measures will hand you a\nfinding that is not there.
![\"PostgreSQL\"](\"https://development-wec.wiline.com/docs/assets/images/Postgresql_elephant.svg-62fedb3464ae364ae4e342c86b474ed6.png\")
+You have two agents. One books appointments. One handles billing.
\nA ticket arrives that needs both: reschedule my install, and my bill looks wrong.\nSo you send it to both.
\nThe first one books Tuesday. The second one sees the account is past due and\nfreezes it. Both finish at almost the same moment, and both save what they decided.
\nOnly one of them is saved. Which one? Whichever finished first ā which comes down to\nhow slow an API call was that day. So you book an appointment on a frozen account,\nor freeze an account you just promised an engineer to. Afterwards it looks like one\nclean decision was made.
\nPart 1 built one agent that runs\nits steps in a fixed order. This post has several: a supervisor that picks who works\non what, an agent that passes the job to another one halfway through, and two agents\nput in each other's way on purpose ā to find out what LangGraph does when they\ndisagree.
\n\nVerified versions: Python 3.10.12 Ā· langgraph 1.2.11 Ā·\nlanggraph-checkpoint 4.2.0 Ā· langgraph-checkpoint-postgres 3.1.2 Ā·\npsycopg 3.3.4 Ā· langchain-openai 1.6.0 Ā· langfuse 4.14.5 against Langfuse\nserver v3.205.1 OSS Ā· image postgres:17. The lg-checkpoints container and\nvirtualenv from Part 1 are reused unchanged.
Two agents means something has to choose between them. That thing is called a\nsupervisor, and it is just another agent whose only job is picking the next one.
\nPart 1 wired its steps together in advance with add_edge ā step one, then two,\nthen three. A supervisor cannot do that, because it does not know where the work\nis going until it reads it.
Instead the node returns a Command:
from langgraph.types import Command
def supervisor(state: State) -> Command[Literal[\"scheduler\", \"billing\"]]:
nxt = \"billing\" if \"invoice\" in state[\"ticket\"].lower() else \"scheduler\"
print(f\"supervisor routing to {nxt}\", flush=True)
return Command(goto=nxt, update={\"completed\": [f\"supervisor->{nxt}\"]})
Command does two jobs at once: goto names the next node, and update carries\na state change with it. There is no add_edge from supervisor to anything ā\nrouting happens at runtime, from inside the function.
Save the whole thing as lg_super.py:
import sys
from operator import add
from typing import Annotated, Literal
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.types import Command
DB_URI = \"postgresql://langgraph:changeme@127.0.0.1:5434/checkpoints\"
THREAD = sys.argv[1] if len(sys.argv) > 1 else \"sup-1\"
TICKET = sys.argv[2] if len(sys.argv) > 2 else \"reschedule my install\"
class State(TypedDict):
ticket: str
completed: Annotated[list[str], add]
def supervisor(state: State) -> Command[Literal[\"scheduler\", \"billing\"]]:
nxt = \"billing\" if \"invoice\" in state[\"ticket\"].lower() else \"scheduler\"
print(f\"supervisor routing to {nxt}\", flush=True)
return Command(goto=nxt, update={\"completed\": [f\"supervisor->{nxt}\"]})
def scheduler(state: State):
print(\"scheduler running\", flush=True)
return {\"completed\": [\"scheduler\"]}
def billing(state: State):
print(\"billing running\", flush=True)
return {\"completed\": [\"billing\"]}
builder = StateGraph(State)
builder.add_node(\"supervisor\", supervisor)
builder.add_node(\"scheduler\", scheduler)
builder.add_node(\"billing\", billing)
builder.add_edge(START, \"supervisor\")
builder.add_edge(\"scheduler\", END)
builder.add_edge(\"billing\", END)
if \"--draw\" in sys.argv:
print(builder.compile().get_graph().draw_mermaid())
raise SystemExit
with PostgresSaver.from_conn_string(DB_URI) as cp:
cp.setup()
graph = builder.compile(checkpointer=cp)
config = {\"configurable\": {\"thread_id\": THREAD}}
print(\"FINAL:\", graph.invoke({\"ticket\": TICKET, \"completed\": []}, config), flush=True)
The --draw branch prints the graph and exits before touching Postgres, so you can\ninspect the topology without a database. It earns its place in the next section.
~/.venv/bin/python ~/lg_super.py sup-1 \"reschedule my install\"
supervisor routing to scheduler
scheduler running
FINAL: {'ticket': 'reschedule my install', 'completed': ['supervisor->scheduler', 'scheduler']}
And the other branch, to show the routing is data-dependent rather than hardcoded:
\n~/.venv/bin/python ~/lg_super.py sup-2 \"please fix the invoice on my account\"
supervisor routing to billing
billing running
FINAL: {'ticket': 'please fix the invoice on my account', 'completed': ['supervisor->billing', 'billing']}
That return annotation looks like ordinary typing you could drop:
\ndef supervisor(state: State) -> Command[Literal[\"scheduler\", \"billing\"]]:
Drop it and the code runs identically ā same routing, same output. So it is\noptional. Except it is not.
\nsed 's/ -> Command\\[Literal\\[\"scheduler\", \"billing\"\\]\\]//' ~/lg_super.py > ~/lg_super_noann.py
Ask both versions to draw themselves:
\n~/.venv/bin/python ~/lg_super.py --draw
~/.venv/bin/python ~/lg_super_noann.py --draw
With the annotation, the two possible routes are there as dotted conditional\nedges:
\n__start__ --> supervisor;
supervisor -.-> billing;
supervisor -.-> scheduler;
billing --> __end__;
scheduler --> __end__;
Without it, they vanish, and the graph states something false:
\n__start__ --> supervisor;
supervisor --> __end__;
scheduler and billing are still declared as nodes, and now nothing reaches\nthem. The diagram says the supervisor goes straight to the end.
![\"Two](\"https://development-wec.wiline.com/docs/assets/images/lg2-graphs-522fc91da9a7554bb7444d68736b3ee7.png\")
\nFigure 1. Same code, one annotation apart. The lower graph is wrong.
The reason is mechanical: goto is decided inside the function body, where\nnothing can inspect it. The annotation is the only declaration of where a\nsupervisor is allowed to send work. Omit it and your code still works while every\ndiagram, and anything else reading the topology, quietly lies.
draw_ascii() needs a package LangGraph does not shipget_graph().draw_ascii() raises\nImportError: Install grandalf to draw graphs: 'pip install grandalf'.\ndraw_mermaid() has no extra dependency and is what the examples above use.
A supervisor decides once, at the start. That is a problem when the discovery\nhappens later.
\nTake a ticket asking to reschedule an install, on an account that turns out to be\npast due. The supervisor sees a scheduling request and routes accordingly. The\nscheduler starts work, reads the account, and finds a problem billing owns.
\nThe scheduler can route too ā it is a node, and nodes return Command.
Copy lg_super.py to lg_handoff.py, replace scheduler with the version below,\nand delete the builder.add_edge(\"scheduler\", END) line ā the node decides its own\nexit now:
def scheduler(state: State) -> Command[Literal[\"billing\", \"__end__\"]]:
if \"past due\" in state[\"ticket\"].lower():
print(\"scheduler: account is past due, handing off to billing\", flush=True)
return Command(goto=\"billing\", update={\"completed\": [\"scheduler(handoff)\"]})
print(\"scheduler: booking it\", flush=True)
return Command(goto=END, update={\"completed\": [\"scheduler(booked)\"]})
~/.venv/bin/python ~/lg_handoff.py handoff-3 \"reschedule my install, account is past due\"
~/.venv/bin/python ~/lg_handoff.py handoff-4 \"reschedule my install\"
supervisor -> scheduler
scheduler: account is past due, handing off to billing
billing running
FINAL: {'ticket': '...past due', 'completed': ['supervisor', 'scheduler(handoff)', 'billing']}
supervisor -> scheduler
scheduler: booking it
FINAL: {'ticket': 'reschedule my install', 'completed': ['supervisor', 'scheduler(booked)']}
![\"Both](\"https://development-wec.wiline.com/docs/assets/images/lg2-handoff-4bb2edcb710dc048a8d846ee980848c2.png\")
\nFigure 2. The supervisor never knew billing would be involved.
Two details worth keeping. goto=END works from inside a Command, so a node can\nfinish the run itself. And \"__end__\" is the string form of END in that\nLiteral ā the annotation needs the literal value, not the constant.
This is the difference between the two published patterns. A supervisor decides\nfrom the outside and needs to know every precondition up front. A handoff lets\nthe agent doing the work redirect once it learns something. Most real systems want\nboth, which is what this graph has.
\nSo far the supervisor picks one agent. Sometimes you want both ā check the calendar\nand the account at the same time, rather than waiting for one before starting the\nother. goto takes a list for exactly that:
return Command(goto=[\"scheduler\", \"billing\"], update={\"completed\": [\"supervisor\"]})
Both run in the same super-step. Which is where it gets interesting, because now\nthey can disagree.
\nCopy lg_super.py to lg_conflict.py. Give the state two kinds of field:
class State(TypedDict):
completed: Annotated[list[str], add] # has a reducer
decision: str # no reducer
And have each agent write a different value to the un-reduced one:
\ndef scheduler(state: State):
return {\"completed\": [\"scheduler\"], \"decision\": \"book tuesday\"}
def billing(state: State):
return {\"completed\": [\"billing\"], \"decision\": \"refund first\"}
~/.venv/bin/python ~/lg_conflict.py conflict-2
supervisor sending work to BOTH
billing deciding
scheduler deciding
Traceback (most recent call last):
...
File \".../langgraph/pregel/_loop.py\", line 692, in after_tick
self.updated_channels = apply_writes(
File \".../langgraph/channels/last_value.py\", line 64, in update
raise InvalidUpdateError(msg)
langgraph.errors.InvalidUpdateError: At key 'decision': Can receive only one value
per step. Use an Annotated key to handle multiple values.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg2-conflict-906aa9ed90d5c25e9021f93fef593bcf.png\")
\nFigure 3. Both agents finished. The graph refused to merge their answers.
Read the trace carefully, because three things in it matter.
\nBoth agents ran. billing deciding and scheduler deciding both printed. The\nwork completed; the failure came afterwards, in after_tick ā apply_writes. This\nis not two agents colliding mid-flight. It is the graph refusing to reconcile them\nat the step boundary.
completed was fine. Both agents appended to it in the same step and nothing\ncomplained, because add says what two values mean. Only decision failed.
LangGraph does not pick a winner. No last-write-wins, no silent choice. Which\nis the right behaviour ā the alternative is a booking agent that sometimes books\nTuesday and sometimes issues a refund depending on which task happened to finish\nfirst.
\nNote the order too: billing printed before scheduler, the reverse of the goto\nlist. Parallel branches finish in whatever order they finish.
The fix is not to prevent the disagreement. It is to say what a disagreement\nmeans. Copy lg_conflict.py to lg_resolve.py and add the reducer:
PRIORITY = {\"refund first\": 2, \"book tuesday\": 1}
def prefer_higher_priority(old: str, new: str) -> str:
\"\"\"The policy: a billing hold outranks a scheduling decision.\"\"\"
winner = max([old, new], key=lambda v: PRIORITY.get(v, 0))
print(f\" reducer: '{old}' vs '{new}' -> '{winner}'\", flush=True)
return winner
class State(TypedDict):
completed: Annotated[list[str], add]
decision: Annotated[str, prefer_higher_priority]
~/.venv/bin/python ~/lg_resolve.py resolve-2
reducer: '' vs '' -> ''
supervisor sending work to BOTH
billing deciding
scheduler deciding
reducer: '' vs 'refund first' -> 'refund first'
reducer: 'refund first' vs 'book tuesday' -> 'refund first'
FINAL: {'completed': ['supervisor', 'billing', 'scheduler'], 'decision': 'refund first'}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg2-resolve-e9d09a07776b7e33a95a0f951453e628.png\")
\nFigure 4. The print inside the reducer makes the merge visible.
Same graph, same disagreement, no error. And the trace shows three things the\ndocumentation does not spell out:
\nThe reducer runs pairwise, folded left. Not once with both values ā twice,\naccumulating. First it merges the existing state with billing's answer, then merges\nthat result with scheduler's.
\nIt must therefore be order-independent. Since branches finish in arbitrary\norder, a reducer like \"take the newest\" produces different answers on different\nruns. max by priority does not.
It fires before anything runs. That first reducer: '' vs '' -> '' is the\ninitial state being written, merging the default with the decision: \"\" passed to\ninvoke. Your reducer sees empty input and must tolerate it.
Part 1 showed a killed run resuming from its last completed node. Parallel work\nraises a sharper question: if one branch finishes and its sibling dies, is the\nfinished work lost?
\nCopy lg_conflict.py to lg_partial.py. Drop the decision field, add\nRESUME = \"--resume\" in sys.argv, pass None if RESUME else {...} to invoke as\nin Part 1, and replace the two specialists with one instant node and one slow\nenough to kill:
def fast(state: State):
print(\"fast running ā finishes immediately\", flush=True)
return {\"completed\": [\"fast\"]}
def slow(state: State):
print(\"slow running ā 30s window, KILL ME HERE\", flush=True); time.sleep(30)
return {\"completed\": [\"slow\"]}
~/.venv/bin/python ~/lg_partial.py partial-2
Press Ctrl+C during the window. It takes two presses, and the run ends on\nsomething that looks much worse than a KeyboardInterrupt:
File \".../langgraph/pregel/_runner.py\", line 613, in commit
self.put_writes()(task.id, task.writes)
...
RuntimeError: cannot schedule new futures after shutdown
That is fast's checkpoint write being committed to an executor that has already\nshut down. It reads like data loss.
It is not:
\n~/.venv/bin/python ~/lg_partial.py partial-2 --resume
BEFORE values={'completed': ['supervisor', 'fast']} next=('slow',)
slow running ā 30s window, KILL ME HERE
FINAL: {'completed': ['supervisor', 'fast', 'slow']}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg2-partial-13329c0354dca69472571333fd73fb02.png\")
\nFigure 5. fast recovered, did not re-run, and next names only the branch that died.
fast is in the recovered state, appears exactly once in the final result, and\nnext=('slow',) names only the branch that never finished. The write had already\nlanded in checkpoint_writes ā the same table that stopped step_one re-running\nin Part 1 now protects a completed sibling.
Worth documenting precisely because the error suggests the opposite. A\nRuntimeError in a shutdown path is noise; the state is the thing to check.
Every routing decision so far has been \"invoice\" in ticket ā keyword matching.\nThat is the approach we measured coming apart on\nthe gateway's complexity router, where a\nplural is enough to miss a keyword entirely.
So let the model decide. Copy lg_super.py to lg_traced.py ā same graph, one\nnode changed:
llm = ChatOpenAI(
model=\"gemma4\",
base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"],
temperature=0,
)
def supervisor(state: State) -> Command[Literal[\"scheduler\", \"billing\"]]:
r = llm.invoke(
\"Route this support ticket to exactly one team. \"
\"Answer with one word, either scheduler or billing, nothing else.\\n\\n\"
f\"Ticket: {state['ticket']}\"
)
raw = (r.content or \"\").strip().lower()
nxt = \"billing\" if \"billing\" in raw else \"scheduler\"
print(f\"supervisor: model said {raw!r} -> {nxt}\", flush=True)
return Command(goto=nxt, update={\"completed\": [f\"supervisor->{nxt}\"]})
Credentials come from the environment, which the gateway's .env already holds:
set -a; . ~/llm-gateway/.env; set +a; ~/.venv/bin/python ~/lg_traced.py traced-1
The ticket is deliberately ambiguous ā \"my bill looks wrong and I need a new\ninstall date\" mentions both:
\nsupervisor: model said 'billing' -> billing
billing running
FINAL: {'ticket': 'my bill looks wrong and I need a new install date', 'completed': ['supervisor->billing', 'billing']}
One word, no wrapper sentence, and the parse held. It picked billing ā defensible,\nsince an unpaid account gates the install, which is the same conclusion the handoff\nreached earlier by reading the account.
\nAsking a model to choose is not free, and the bill is not where you would look for\nit. Turning on tracing shows where the time and the tokens went. It is one line, as\nin Part 1 ā CallbackHandler() reads its credentials from the environment and goes\nin the same config dict as the thread id:
from langfuse.langchain import CallbackHandler
handler = CallbackHandler()
config = {\"configurable\": {\"thread_id\": THREAD}, \"callbacks\": [handler]}
![\"Langfuse](\"https://development-wec.wiline.com/docs/assets/images/lg2-trace-771785326bcdaf3d0132233cc7bdf9fc.png\")
\nFigure 6. Three things boxed: what routing cost, the decision as data, and the payload arriving.
The span tree is the graph:
\nLangGraph 8.66s
supervisor 8.53s
ChatOpenAI 8.49s 52 ā 412 (Ī£ 464)
billing 0.01s
Three findings sit in that frame.
\nA one-word answer cost 412 output tokens. Confirmed against the API rather\nthan read off the screen:
\nset -a; . ~/llm-gateway/.env; set +a
curl -s -u \"$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY\" \\
\"$LANGFUSE_HOST/api/public/observations?limit=8\" \\
| jq '.data[] | select(.name==\"ChatOpenAI\") | {model, usage}'
{
\"model\": \"gemma4\",
\"usage\": {
\"unit\": \"TOKENS\",
\"input\": 52,
\"output\": 412,
\"total\": 464
}
}
The visible answer is seven characters. The bill is 412 output tokens. Whatever the\nmodel generated before settling, you paid for ā and a routing call looks like the\ncheapest thing in the graph.
\nThe supervisor is the expensive node, not the specialists. billing took 6ms\nand no tokens. The router took 8.49s and 464. In a multi-agent graph, the\norchestration decisions are the cost centre.
The routing decision is recorded as data. The supervisor's output in the trace\nis the serialised Command:
{\"graph\": null, \"update\": {\"completed\": [\"supervisor->billing\"]}, \"resume\": null, \"goto\": \"billing\"}
And billing's input contains [\"supervisor->billing\"] ā the update from that\nCommand arriving as the receiving agent's state. Handoff, cost and payload in one\nframe.
scheduler appears nowhere in this trace ā not as a skipped span, not as a\ndisabled node. The Graph panel draws __start__ ā supervisor ā billing ā __end__.\nYou cannot tell from a trace which alternatives existed, only which one ran, which\nis a second reason the missing Literal annotation matters: the drawn graph is the\nonly place your alternatives are written down.
Part 3: the tools an agent is allowed to call. Everything above trusts each node\nto do only what it should ā the next post puts the tools behind a gateway with\nper-tool permissions, so an agent's capabilities are granted rather than assumed.
\nImportError: Install grandalf to draw graphsādraw_ascii() needs a package LangGraph does not depend on. Use draw_mermaid().
__end__āThe routing node is missing its -> Command[Literal[...]] return annotation. The\ncode still runs; only the topology is wrong.
InvalidUpdateError: At key 'x': Can receive only one value per stepāTwo nodes wrote the same key in one super-step and that key has no reducer. Either\ngive it one with Annotated[T, fn], or stop dispatching both nodes in parallel.
RuntimeError: cannot schedule new futures after shutdown after Ctrl+CāA completed branch's checkpoint write raced the executor shutting down. Check\ngraph.get_state(config).values before assuming anything was lost ā in testing the\nwrite had landed every time.
It is not order-independent. Parallel branches finish in arbitrary order, so a\nreducer that depends on which value arrives second is non-deterministic by\nconstruction.
\n+Your agent has been working on a customer's request for forty seconds. It has read\nthe ticket, pulled the account, and checked three engineers' calendars. It is about\nto book the appointment.
\nThen the process dies. A deploy going out, the host running out of memory, someone\nrestarting the container ā it does not matter which.
\nThe agent does not pick up where it left off, because there is nothing to pick up\nfrom. Everything it learned lived in variables inside a process that no longer\nexists. The customer is still waiting. Run it again and you pay for all that work a\nsecond time. And the part that should worry you most: nobody can say whether the\nappointment was booked in the last second before it died.
\nAn agent is a model in a loop. As an ordinary Python script, that loop is exactly as\nfragile as the process holding it.
\nHere we build one that saves its state to Postgres after every step, so a crash\ncosts nothing.
\n\nVerified versions: Python 3.10.12 Ā· langgraph 1.2.11 Ā·\nlanggraph-checkpoint 4.2.0 Ā· langgraph-checkpoint-postgres 3.1.2 Ā·\npsycopg 3.3.4 Ā· psycopg-pool 3.3.1 Ā· langchain-openai 1.6.0 Ā·\nlangfuse 4.14.5 against Langfuse server v3.205.1 OSS Ā· image postgres:17.
What you want is a resumable run: each step's result written somewhere durable\nthe moment it finishes, so a new process can carry on from there.
\nThat is what a checkpointer does.
\nA loop with a try/except and a few database writes could do this for four fixed\nsteps. It stops working the moment the agent picks its own path: when the model\nchooses the next tool, \"where are we?\" has no answer you can write down.
LangGraph makes you declare the work as a graph ā named nodes, explicit edges.\nThat feels like ceremony until you want durability.
\n| A loop | A graph |
|---|---|
| Steps are implicit in control flow | Steps are named nodes |
| \"Where am I?\" has no answer | Position is a value you can read |
| State lives in local variables | State is a declared object |
| Nothing to save | There is an obvious moment to save: node boundaries |
The runtime knows when a node starts and ends, so it has an obvious moment to save.\nThe state is a declared object, so there is something definite to write.
\nThat is what \"a deterministic state machine around a non-deterministic model\" means:\nthe output is unpredictable, the next step is not. More on that debate in\n\"Loop Engineering Is Dead\".
\nThree properties the usual examples do not have:
\npython3 -m venv .venv
source .venv/bin/activate
pip install \"langgraph\" \"langgraph-checkpoint-postgres\" \"psycopg[binary,pool]\"
| Package | What it does | Why you need it here |
|---|---|---|
langgraph | The graph runtime ā nodes, edges, state | The orchestration itself |
langgraph-checkpoint-postgres | Writes state to Postgres | The point of this tutorial |
psycopg[binary,pool] | Postgres driver + connection pool | PostgresSaver requires a pool |
The middle row is worth pausing on. LangGraph's default persistence is\nin-memory. Durability is opt-in, in a separate package you have to know to\ninstall. That is precisely why so many published examples quietly lose state: they\nnever installed this, and nothing warned them.
\nThree names pull in more than three packages:
\n./.venv/bin/pip list | grep -iE \"langgraph|psycopg\" && python3 --version
langgraph 1.2.11
langgraph-checkpoint 4.2.0
langgraph-checkpoint-postgres 3.1.2
langgraph-prebuilt 1.1.0
langgraph-sdk 0.4.3
psycopg 3.3.4
psycopg-binary 3.3.4
psycopg-pool 3.3.1
Python 3.10.12
langgraph-checkpoint is the one to notice: it is the abstract interface, and\n-postgres is one implementation of it. That split is why swapping Postgres for\nSQLite or Redis later changes one line.
The pool extra is not decoration either. Install plain psycopg and the failure\ncomes at connect time, not install time ā which is a much more confusing place to\nfind it.
![\"pip](\"https://development-wec.wiline.com/docs/assets/images/lg1-versions-21d23b4acd78bb5e757cf1a9f3949f97.png\")
\nFigure 1. Eight packages from three names ā the checkpoint interface and its\nPostgres implementation arrive separately.
docker run -d --name lg-checkpoints \\
-e POSTGRES_USER=langgraph -e POSTGRES_PASSWORD=changeme -e POSTGRES_DB=checkpoints \\
-p 127.0.0.1:5434:5432 postgres:17
Three deliberate decisions:
\nA dedicated database. Checkpoints are write-heavy ā a row per node transition,\nper thread, forever. Mixed into your application database they become a table you\nare afraid to truncate. Separate, they are disposable.
\n127.0.0.1:5434:5432, not -p 5434:5432. The short form binds every\ninterface, and Docker writes its own iptables rules that bypass UFW ā so a\nfirewall that looks correct is not protecting this port. Covered in\nHarden Docker networks.
The left number is the host port, the right the container's. On an empty instance\nuse 5432:5432. If you already self-host anything with a database, that port is\nprobably taken ā pick any free one.
postgres:17 pinned. latest means a reader six months from now runs\nsomething you never tested.
Running is not the same as ready:
\ndocker exec lg-checkpoints pg_isready -U langgraph -d checkpoints
/var/run/postgresql:5432 - accepting connections
A container reports Up the instant it starts, while Postgres inside is still\ninitialising. Connecting too early gives a connection error that looks like a\nconfiguration problem and is not.
No LLM in this step. If something fails now, it is the checkpointer's fault and\nnot the model's ā and separating those two is most of what debugging an agent\nconsists of.
\nfrom operator import add
from typing import Annotated
from typing_extensions import TypedDict
class State(TypedDict):
completed: Annotated[list[str], add]
This declares what the graph remembers. TypedDict gives the shape; the\nAnnotated[..., add] part is the interesting half.
By default, when a node returns a value for a key, that value replaces what\nwas there. Annotated[list[str], add] attaches a reducer ā a function\ncombining the old value with the new one instead of overwriting it. With\noperator.add on a list, that means append.
| Declaration | step_one returns [\"a\"], then step_two returns [\"b\"] |
|---|---|
completed: list[str] | [\"b\"] ā the first result is lost |
completed: Annotated[list[str], add] | [\"a\", \"b\"] ā accumulates |
Get this wrong and your state silently forgets everything except the last node.
\ndef step_one(state: State):
print(\"step_one running\", flush=True); time.sleep(2)
return {\"completed\": [\"step_one\"]}
def step_two(state: State):
print(\"step_two running ā 30s window, KILL ME HERE\", flush=True); time.sleep(30)
return {\"completed\": [\"step_two\"]}
A node is a plain function. It receives the current state and returns a partial\nupdate ā only the keys it changed, not the whole object. The runtime merges that\nupdate using the reducers you declared.
\nThe thirty-second sleep in step_two stands in for a slow step ā a model call, an\nexternal API, a batch job. It is also what gives you time to kill the process on\ncue in the next step.
builder = StateGraph(State)
for name, fn in [(\"step_one\", step_one), (\"step_two\", step_two), (\"step_three\", step_three)]:
builder.add_node(name, fn)
builder.add_edge(START, \"step_one\")
builder.add_edge(\"step_one\", \"step_two\")
builder.add_edge(\"step_two\", \"step_three\")
builder.add_edge(\"step_three\", END)
StateGraph(State) binds the schema. add_node registers a function under a\nname ā that name is what appears in checkpoints and in traces later. add_edge\ndeclares order, with START and END as the sentinels marking entry and exit.
This graph is a straight line. Branching comes in part 2.
\nwith PostgresSaver.from_conn_string(DB_URI) as checkpointer:
checkpointer.setup()
graph = builder.compile(checkpointer=checkpointer)
config = {\"configurable\": {\"thread_id\": THREAD}}
result = graph.invoke({\"completed\": []}, config)
Four lines, each carrying a trap.
\nwith ... as checkpointer. from_conn_string is decorated @contextmanager\nin the source ā it yields, it does not return. Assigning it directly hands you a\ncontext manager object where you expected a saver, and the error arrives much\nlater than the mistake.
It also opens the connection with autocommit=True, prepare_threshold=0, and\nrow_factory=dict_row. Pass your own connection and you must set those yourself, or\n.setup() can look like it worked and save nothing. Why each is needed is documented\nnowhere ā see issue #4937,\nclosed without an answer.
.setup() is required on first use. It creates the tables and runs\nmigrations.
compile(checkpointer=...). Without this argument the graph runs perfectly and\nsaves nothing. There is no warning. It is the single most likely reason someone\nbelieves checkpointing \"doesn't work\".
thread_id. The identifier for one conversation or one run. Every checkpoint\nis scoped to it, and resuming means passing the same one. It must stay under 255\ncharacters.
Together, that is lg_agent.py:
import sys, time
from operator import add
from typing import Annotated
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.postgres import PostgresSaver
DB_URI = \"postgresql://langgraph:changeme@127.0.0.1:5434/checkpoints\"
THREAD = sys.argv[1] if len(sys.argv) > 1 else \"ticket-1\"
RESUME = \"--resume\" in sys.argv
class State(TypedDict):
completed: Annotated[list[str], add]
def step_one(state: State):
print(\"step_one running\", flush=True); time.sleep(2)
return {\"completed\": [\"step_one\"]}
def step_two(state: State):
print(\"step_two running ā 30s window, KILL ME HERE\", flush=True); time.sleep(30)
return {\"completed\": [\"step_two\"]}
def step_three(state: State):
print(\"step_three running\", flush=True)
return {\"completed\": [\"step_three\"]}
builder = StateGraph(State)
for name, fn in [(\"step_one\", step_one), (\"step_two\", step_two), (\"step_three\", step_three)]:
builder.add_node(name, fn)
builder.add_edge(START, \"step_one\")
builder.add_edge(\"step_one\", \"step_two\")
builder.add_edge(\"step_two\", \"step_three\")
builder.add_edge(\"step_three\", END)
with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
checkpointer.setup()
graph = builder.compile(checkpointer=checkpointer)
config = {\"configurable\": {\"thread_id\": THREAD}}
snap = graph.get_state(config)
print(f\"BEFORE values={snap.values} next={snap.next}\", flush=True)
result = graph.invoke(None if RESUME else {\"completed\": []}, config)
print(\"FINAL:\", result, flush=True)
snap = graph.get_state(config)
print(f\"AFTER values={snap.values} next={snap.next}\", flush=True)
RESUME is what makes --resume work: on a resume the input is None rather\nthan a fresh state, which is covered in the next step.
python lg_agent.py ticket-41
After the first run, four tables exist:
\ndocker exec lg-checkpoints psql -U langgraph -d checkpoints -c '\\dt'
public | checkpoint_blobs | table | langgraph
public | checkpoint_migrations | table | langgraph
public | checkpoint_writes | table | langgraph
public | checkpoints | table | langgraph
| Table | Holds |
|---|---|
checkpoints | One row per node boundary ā the snapshot index |
checkpoint_blobs | The serialised state values |
checkpoint_writes | Pending writes from nodes that finished in a super-step |
checkpoint_migrations | Schema version bookkeeping |
checkpoint_writes is the one that makes crash recovery work, as the next step\nshows.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg1-tables-f26befffa3adb092e0cecaa4126737c2.png\")
\nFigure 2. .setup() writes the schema for you ā no manual DDL.
snap = graph.get_state(config)
print(f\"BEFORE values={snap.values} next={snap.next}\")
get_state returns a StateSnapshot. Two fields matter: values is the state\nright now, and next is the tuple of nodes still to run. next=() means the\nthread is finished or has never started; anything else names what is pending.
next is how you answer \"where is this run?\" ā the question a loop cannot answer.
Start a fresh thread, then press Ctrl+C during the thirty-second window.
\npython lg_agent.py ticket-42
BEFORE values={} next=()
step_one running
step_two running ā 30s window, KILL ME HERE
^CKeyboardInterrupt
![\"KeyboardInterrupt](\"https://development-wec.wiline.com/docs/assets/images/lg1-killed-13e4a54ded6e98bd000ca28160e0eb18.png\")
\nFigure 3. A real crash ā the traceback comes from inside step_two, which therefore never completed.
Now resume the same thread_id:
python lg_agent.py ticket-42 --resume
BEFORE values={'completed': ['step_one']} next=('step_two',)
step_two running ā 30s window, KILL ME HERE
step_three running
FINAL: {'completed': ['step_one', 'step_two', 'step_three']}
AFTER values={'completed': [...]} next=()
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg1-resumed-060c0f536dc33b295570816403af58d0.png\")
\nFigure 4. The BEFORE line is the proof ā state recovered from Postgres by a brand-new process.
Read that first line closely, because it is the whole tutorial:
\nvalues shows step_one's result, recovered from Postgres by a process that\nhad not existed when it was producednext=('step_two',) ā the graph knows exactly which node did not finishstep_one did not run again. It executed once across three invocationsThat last point is what checkpoint_writes buys: LangGraph keeps the completed\nwrites from a super-step, so resuming does not repeat work that succeeded.
The resume call itself is:
\ngraph.invoke(None, config)
None as input means \"no new input ā continue from the checkpoint.\" Worth\nknowing: this is documented on the interrupts page for static breakpoints, but not\non the persistence or durable-execution pages, which is where you look when your\nprocess has just died. It works; it is simply not written down where you need it.
Kill it a second time and resume again. It works, because reading a checkpoint\ndoes not consume it.
\nA crash is an accidental stop. An approval is a deliberate one ā and once you can\nsurvive the first, the second is nearly free.
\nThe agent is about to write an appointment into a customer's calendar. That is\nirreversible and a real person sees it, so you want someone to check first.
\nWithout durable state, waiting for a human means holding a process open. With state\nin Postgres the agent stops, the process exits, and the approval arrives whenever it\narrives ā ten minutes or Monday morning.
\nfrom langgraph.types import interrupt, Command
def approval(state: State):
decision = interrupt({\"question\": \"Approve step_two?\", \"so_far\": state[\"completed\"]})
return {\"completed\": [\"approval\"], \"approved\": decision}
Save it as lg_agent2.py and run it:
python lg_agent2.py booking-7
interrupt() does two things at once: it stops the graph, and it hands its\nargument out to whoever is calling. That argument is your question to the human ā\nany JSON-serialisable value.
step_one running
RESULT: {'completed': ['step_one'], 'approved': '',
'__interrupt__': [Interrupt(value={'question': 'Approve step_two?',
'so_far': ['step_one']}, id='29f211cf5a73ea78e0f694e1bd38822b')]}
AFTER next=('approval',) interrupts=(Interrupt(...),)
No exception, exit code 0. Note the difference from Step 4: the crash produced a\ntraceback, this produces a clean return.
\nThe pause surfaces in two places, and the difference matters:
\n| Where | Use |
|---|---|
__interrupt__ in the invoke result | The caller that just ran the graph |
snapshot.interrupts | Any other process ā a dashboard rendering the question |
The id correlates an answer back to the right pause, which matters once more\nthan one approval is outstanding.
The process is now gone; the state is in Postgres. An hour or a week later:
\npython lg_agent2.py booking-7 yes
which calls:
\ngraph.invoke(Command(resume=\"yes\"), config)
approval resumed with: yes
step_two running (approved=yes)
AFTER next=() interrupts=()
Command(resume=...) is different from invoke(None, ...). None says \"carry\non\"; Command(resume=X) says \"carry on, and interrupt() should return X.\"\nThat value came from the command line, through interrupt()'s return, into state,\nand was read by the next node.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg1-interrupt-ea585f38dd6726bebaeeff4db224460a.png\")
\nFigure 5. Pause and resume ā two separate processes, one run.
On resume, LangGraph re-executes the interrupted node from the top. This time\ninterrupt() returns your value instead of pausing ā but everything above it in\nthat function has already run a second time.
Put a print before the interrupt, then run a fresh thread and approve it:
\npython lg_agent2.py booking-8
python lg_agent2.py booking-8 approve
It appears in both runs, for a single approval:
\nrun 1: step_one running
SIDE EFFECT ā before interrupt
RESULT: {... '__interrupt__': [Interrupt(...)]}
run 2: SIDE EFFECT ā before interrupt
approval resumed with: approve
step_two running (approved=approve)
![\"The](\"https://development-wec.wiline.com/docs/assets/images/lg1-double-side-effect-89c6352b4d9d897ec50519025547fca8.png\")
\nFigure 6. One approval, two executions of everything above the interrupt.
If that line were send_email() or charge_card(), one approval charges the\ncustomer twice. Nothing raises. Final state is correct. Only the side effect\nduplicated ā which is why this is easy to ship and hard to notice.
Put nothing before interrupt() in that node. Move side effects after it, or\ngive the interrupt a node of its own that does nothing else. The same reasoning\napplies to crash resume: a node can always run more than once, so anything with an\nexternal consequence should be idempotent or isolated.
Two more packages: one to talk to the model, one to trace it.
\npip install langchain-openai langfuse
llm = ChatOpenAI(
model=\"Qwen2.5-3B-Instruct\",
base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"],
temperature=0,
)
ChatOpenAI is not OpenAI-specific ā it speaks the OpenAI wire protocol, which\nWEC Inference implements. Pointing base_url at it is the whole integration.\ntemperature=0 keeps runs comparable while you are testing.
The call goes inside a node like any other work:
\ndef write(state: State):
r = llm.invoke(f\"Write one sentence about {state['topic']}.\")
return {\"completed\": [\"write\"], \"draft\": r.content}
You need a Langfuse project and an API key pair for this ā created under\nOrganization ā Project, then Settings ā API Keys, as covered in\nObserve production with Langfuse.
\nTracing attaches per invocation, in the same config dict as the thread id:
\nfrom langfuse.langchain import CallbackHandler
handler = CallbackHandler()
config = {\"configurable\": {\"thread_id\": THREAD}, \"callbacks\": [handler]}
configurable is LangGraph's own settings; callbacks is LangChain's observer\nhook. Langfuse subscribes to node start and end events and builds spans from them\nā which is why the trace mirrors your graph without any extra instrumentation.
Credentials come from the environment ā in SDK v4 CallbackHandler() takes no\narguments, so this is the only way to configure it:
export WEC_API_KEY='...'
export LANGFUSE_PUBLIC_KEY='pk-lf-...'
export LANGFUSE_SECRET_KEY='sk-lf-...'
export LANGFUSE_HOST='http://localhost:3001'
Putting it together as lg_llm.py:
import os, sys
from operator import add
from typing import Annotated
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.postgres import PostgresSaver
from langchain_openai import ChatOpenAI
from langfuse.langchain import CallbackHandler
DB_URI = \"postgresql://langgraph:changeme@127.0.0.1:5434/checkpoints\"
THREAD = sys.argv[1] if len(sys.argv) > 1 else \"draft-1\"
llm = ChatOpenAI(
model=\"Qwen2.5-3B-Instruct\",
base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"],
temperature=0,
)
class State(TypedDict):
topic: str
completed: Annotated[list[str], add]
draft: str
def plan(state: State):
print(\"plan running\", flush=True)
return {\"completed\": [\"plan\"]}
def write(state: State):
r = llm.invoke(f\"Write one sentence about {state['topic']}.\")
print(\"model said:\", r.content, flush=True)
return {\"completed\": [\"write\"], \"draft\": r.content}
builder = StateGraph(State)
builder.add_node(\"plan\", plan)
builder.add_node(\"write\", write)
builder.add_edge(START, \"plan\")
builder.add_edge(\"plan\", \"write\")
builder.add_edge(\"write\", END)
handler = CallbackHandler()
with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
checkpointer.setup()
graph = builder.compile(checkpointer=checkpointer)
config = {\"configurable\": {\"thread_id\": THREAD}, \"callbacks\": [handler]}
result = graph.invoke(
{\"topic\": \"edge computing\", \"completed\": [], \"draft\": \"\"}, config
)
print(\"FINAL:\", result, flush=True)
python lg_llm.py draft-1
LangGraph 2.18s
plan 0.00s
write 2.14s
ChatOpenAI 2.13s 36 -> 37 (73 tokens)
![\"Langfuse](\"https://development-wec.wiline.com/docs/assets/images/lg1-langfuse-trace-8385ab962b94706bb03e2df32a87a0a6.png\")
\nFigure 7. Span tree, graph topology and token counts ā from one callback.
Almost all of it is the model call. That leaves about 40ms for the graph and\ntracing, and the gap stayed near 40ms even on runs that took twice as long. plan,\nwhich calls nothing, reports 0.00s.
The useful conclusion: orchestration is not your latency problem. If an agent\nfeels slow, the graph is not why.
\naddress already in use vs port is already allocated. These look\ninterchangeable and are not. The first means a host process holds the port; the\nsecond means another container does ā Docker's own bookkeeping, before the bind\nis attempted. Diagnose with:
docker ps --format '{{.Names}}\\t{{.Ports}}' && sudo ss -ltnp | grep :543
A failed docker run blocks the name. The container is created, then dies, and\nstill holds its name ā so your retry fails differently than the first attempt did.\ndocker rm <name> before retrying.
ModuleNotFoundError: No module named 'langchain' when importing Langfuse's\nCallbackHandler. Neither LangGraph nor langchain-openai needs the langchain\numbrella package; Langfuse imports it purely as a version sentinel. The fix is\ninstalling a package nothing else in your stack uses.
Two upstream threads are worth knowing apart.\n#9758 reported the import\nfailure in October 2025 and is closed ā but it still reproduces on the SDK\nversion above, so read it as history rather than as a fix that landed.\n#13651 is open and\nproposes the actual remedy: read langchain_core.__version__ instead of requiring\nthe umbrella package at all.
Langfuse disables itself in silence. Miss an environment variable and you get\none warning line, then a completely normal run ā exit code 0, correct output, no\ntraces at all:
\nAuthentication error: Langfuse client initialized without public_key.
Client will be disabled.
In SDK v4 CallbackHandler() accepts no constructor arguments, so environment\nvariables are the only configuration path ā and a missing one does not raise.\nAssert the client is enabled at startup rather than trusting the absence of errors.
Version drift. Most LangGraph + Langfuse material targets Langfuse SDK v3. On\nv4 the update_trace parameter is gone and raises TypeError, and credentials can\nno longer be passed to the constructor.
Part 2: handing work between agents ā and what happens to shared state when they\ndisagree.
\nRelated reading on this site:
\n+Part 3 configured Presidio to mask\nprompts at logging time and deliberately stopped short of proving it. Nothing had\nbeen wired to a logging destination yet, so there was no trace to inspect.
\nThis post wires one up, finds <PERSON> where it should be ā and then finds the\ncustomer's real email address sitting a few lines below it, in the model's reply.
Everything in this post follows from where those two hooks sit. post_call fires\nbefore the arrow back to the caller, so masking there rewrites the answer they\nreceive. async_logging_hook fires after it, on the branch to the logger only.
Langfuse is already running in this series; if you followed\nthe observability tutorial you\nhave an instance. Create a project for the gateway, then take its two keys from\nSettings ā API Keys.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw4-langfuse-keys-0868edc9108438174e185d84a9801382.png\")
Figure 1. A separate project keeps gateway traffic away from whatever else\nyou are tracing.
\nAdd three variables to the gateway's .env:
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=http://your-host:3001
Langfuse's own setup screen prints a .env block containing\nLANGFUSE_BASE_URL. The gateway does not read that. It reads:
os.getenv(\"LANGFUSE_HOST\", \"https://cloud.langfuse.com\")
Copy their snippet verbatim and your host is ignored in favour of the public\ncloud endpoint ā where your self-hosted keys will not authenticate. Traces go\nnowhere, and nothing errors.
Then turn the callback on:
\nlitellm_settings:
drop_params: true
success_callback: [\"langfuse\"]
failure_callback: [\"langfuse\"]
docker compose restart litellm
Send a request after that and the logs say:
\nLangfuse client is disabled since no public_key was provided as a parameter
or environment variable 'LANGFUSE_PUBLIC_KEY'.
The keys are in .env. They are not in the container:
docker exec llm-gateway printenv | grep -c LANGFUSE
0
restart does not reload env_filedocker compose restart restarts the process inside the container that\nalready exists, with the environment it was created with. New variables in\n.env are not picked up.
The config.yaml change did take effect, because that is a bind-mounted file\nread at startup. So the gateway looks correctly configured ā callbacks\ninitialised, no errors ā while having no credentials at all.
Use docker compose up -d, which recreates the container.
docker compose up -d
docker exec llm-gateway printenv | grep -c LANGFUSE
3
![\"printenv](\"https://development-wec.wiline.com/docs/assets/images/gw4-recreate-env-8aff004f8b60f6c931690f76127937bf.png\")
Figure 2. After up -d, the variables are in the container.
Wait for Application startup complete, then send a prompt carrying a name and\nan address:
source .env && curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"What is the first name of Maria Alvarez, and what is the domain of maria.alvarez@example.com? Answer in one short line.\"}],\"max_tokens\":40}' \\
| jq -r '.choices[0].message.content'
The first name of Maria Alvarez is Maria; the domain of maria.alvarez@example.com is example.com.
That answer is the control. The model could not have produced \"Maria\" from a\nplaceholder, so the live request reached it unmasked ā exactly what\nlogging_only promises.
Now the trace:
\n![\"A](\"https://development-wec.wiline.com/docs/assets/images/gw4-trace-input-masked-f167939e2680fdb4984817374cb86a8e.png\")
Figure 3. Input masked. Output not.
\nWhat is the first name of <PERSON>, and what is the domain of <EMAIL_ADDRESS>?
The first name of Maria Alvarez is Maria; the domain of maria.alvarez@example.com is example.com.
Part 3's job is done ā the prompt is masked in the log while the model saw the\nreal text. And the PII is in the trace anyway, because the model repeated it\nback.
\nThat is not an edge case. Repeating the customer's name is what a support\nassistant is for.
\nThe gateway's own database stores the same request differently again:
\ncurl -s http://127.0.0.1:4000/spend/logs \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" | jq '.[0] | {messages, response}'
\"messages\": {}
\"response\": { ... \"content\": \"The first name of Maria Alvarez is Maria;
the domain of maria.alvarez@example.com is example.com.\" ... }
No prompt at all, and the completion in full. Two stores, two shapes, the same\nhole in both.
\nPart 3 set presidio_filter_scope: input. The obvious fix is to scan both\ndirections:
presidio_filter_scope: both
Restart, send the identical request, and read the response the caller gets:
\nThe first name of <PERSON> is <PERSON>; the domain of <EMAIL_ADDRESS> is <URL>.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw4-response-masked-5e5f2e3cd767089449d5067827a9f518.png\")
Figure 4. The caller's own terminal, not a log viewer.
\nThat is not the log. That is the answer returned to the client.
\nmode: logging_only is still set. It did not help, because it never reaches the\noutput path ā the documentation for the scope setting says so directly:
\n\nUse
\npresidio_filter_scope: output(orboth) when you want Presidio to\nactively scan and mask the model's response before it reaches the user.
Masking the live response is the documented purpose of an output scope. What is\nundocumented is the combination: nothing describes what happens when you set\nlogging_only and an output scope, and the code resolves it in favour of the\nlive path.
In guardrail_initializers.py:
if run_output:
output_callback = _make_presidio_callback(
apply_to_output=True,
event_hook=GuardrailEventHooks.post_call.value, # hard-coded
output_parse_pii=False,
)
The mode you configured is passed to the input callback. The output callback\ngets post_call regardless.
A masked reply has two possible causes: the response was masked on the way out,\nor the model received placeholders and answered honestly about them. They look\nidentical.
Ask for something that encodes the real value without containing it ā a single\nletter is not PII, so masking cannot hide it:
curl ... -d '{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with ONLY the first letter of the first name of Maria Alvarez.\"}],\"max_tokens\":200}'
MĀŖ
M. The model had the real name. Only the response was rewritten.
So the two settings give you a choice, and neither is the one you want:
\n| Logged input | Logged output | Caller's response | |
|---|---|---|---|
filter_scope: input | masked | raw PII | untouched |
filter_scope: both | masked | masked | masked |
This has been reported.\nIssue #30447 ā \"logging_only\nPresidio guardrail corrupts user-facing response\" ā was filed on 15 June 2026\nand closed the next day, with no discussion, alongside a pull request titled\n\"fix(presidio): don't mask the live request when guardrail is logging_only\".\nThe report was about the response. The fix addressed the request.\n#35951 is still open on a\nrelated path.
\nThe gap exists because post_call runs while the response is still travelling\nto the caller. There is a later hook that does not.
async_logging_hook fires after the model has answered and before the loggers\nrun. It receives the request and the result, and whatever it returns is what\ngets logged. The caller already has their response by then.
That hook is documented, under\nScrub Logged Data ā but\nthe example there is a placeholder that replaces every message with the literal\nstring MASK_THIS_ASYNC_VALUE. It shows the hook exists; it does not mask\nanything, it touches only kwargs[\"messages\"], and it mutates in place. What\nfollows uses the same extension point and adds the parts that make it work:\nreal Presidio calls, all three places PII hides, and a copy so the caller's\nresponse survives.
The snippet on that page ends return kwargs, responses, but the parameter is\nnamed result. responses is undefined, so copying it verbatim raises\nNameError.
Create scrubber.py next to config.yaml:
import copy
import os
from typing import Any, Tuple
import httpx
from litellm.integrations.custom_logger import CustomLogger
ANALYZER = os.getenv(\"PRESIDIO_ANALYZER_API_BASE\", \"http://presidio-analyzer:3000\")
ANONYMIZER = os.getenv(\"PRESIDIO_ANONYMIZER_API_BASE\", \"http://presidio-anonymizer:3000\")
async def _mask(text: str) -> str:
if not text or not text.strip():
return text
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f\"{ANALYZER}/analyze\", json={\"text\": text, \"language\": \"en\"})
found = r.json()
if not found:
return text
r = await client.post(
f\"{ANONYMIZER}/anonymize\",
json={\"text\": text, \"analyzer_results\": found},
)
return r.json()[\"text\"]
class PresidioLogScrubber(CustomLogger):
async def async_logging_hook(
self, kwargs: dict, result: Any, call_type: str
) -> Tuple[dict, Any]:
if call_type not in (\"completion\", \"acompletion\"):
return kwargs, result
kwargs = copy.deepcopy(kwargs)
for message in kwargs.get(\"messages\") or []:
if isinstance(message.get(\"content\"), str):
message[\"content\"] = await _mask(message[\"content\"])
slo = kwargs.get(\"standard_logging_object\")
if isinstance(slo, dict):
for message in slo.get(\"messages\") or []:
if isinstance(message, dict) and isinstance(message.get(\"content\"), str):
message[\"content\"] = await _mask(message[\"content\"])
response = slo.get(\"response\")
if isinstance(response, dict):
for choice in response.get(\"choices\") or []:
message = (choice or {}).get(\"message\") or {}
if isinstance(message.get(\"content\"), str):
message[\"content\"] = await _mask(message[\"content\"])
# `result` is the object the caller already holds. Copy before masking.
logged_result = copy.deepcopy(result)
for choice in getattr(logged_result, \"choices\", None) or []:
message = getattr(choice, \"message\", None)
if message is not None and isinstance(getattr(message, \"content\", None), str):
message.content = await _mask(message.content)
return kwargs, logged_result
instance = PresidioLogScrubber()
Three details carry the whole thing.
\nPII hides in three places, not one. kwargs[\"messages\"] is the prompt.\nkwargs[\"standard_logging_object\"] is what the logging integrations actually\nread ā this is the subject of issue #35951. And result holds the completion.\nMiss any one and something leaks.
result is the caller's object. Mutating it in place reproduces the bug we\nare working around. It gets deep-copied first, and the copy is what we mask and\nreturn.
_mask is async on purpose. More on that below, because getting it wrong\ncosts measurable latency.
Mount the file and register it. It must sit beside config.yaml, because\nget_instance_fn resolves the dotted path relative to the config file's\ndirectory:
volumes:
- ./config.yaml:/app/config.yaml:ro
- ./scrubber.py:/app/scrubber.py:ro
litellm_settings:
drop_params: true
callbacks: [\"scrubber.instance\"]
success_callback: [\"langfuse\"]
Set the built-in guardrail to default_on: false while testing. If both it and\nthe callback are masking, you cannot tell which did the work.
Adding a volume changes the container definition, so this needs a recreate, not\na restart:
\ndocker compose up -d
Send a request and keep the response body:
\ncurl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"Who is Priya Raghunathan and what is the domain of priya.raghunathan@example.net? One short line.\"}],\"max_tokens\":40}' \\
-o response.json
What the caller received:
\njq -r '.choices[0].message.content' response.json
Priya Raghunathan is an AI researcher, and the domain
priya.raghunathan@example.net is likely a personal email address.
What Langfuse stored:
\ncurl -s -u \"$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY\" \\
\"http://your-host:3001/api/public/traces?limit=1\" -o trace.json
jq -r '.data[0].output.content' trace.json
<PERSON> is an AI researcher, and the domain <EMAIL_ADDRESS> is likely a
personal email address.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw4-side-by-side-54fd1e85dfe8ba2b6544c1a470fb77f9.png\")
Figure 5. Same sentence. One real, one masked.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw4-scrubber-working-29c3b666ef2aa43daef02208253f2fd4.png\")
Figure 6. Masked on both sides of the trace, untouched for the caller.
\nReading a rendered UI proves less than checking the payloads, so check them:
\ngrep -c \"Priya Raghunathan\" response.json
grep -c \"Priya Raghunathan\" trace.json
grep -o \"PERSON\" trace.json | wc -l
grep -c \"One short line\" trace.json
1
0
2
1
![\"Four](\"https://development-wec.wiline.com/docs/assets/images/gw4-grep-proof-f38350595a637c78223db0c9f01b0da1.png\")
Figure 7. The receipt: 1 0 2 1.
The real name is in what the caller received. It is not in the trace. The\nplaceholder appears twice ā input and output. And the last line proves both\nfiles describe the same request, which the first three do not establish on\ntheir own.
\nYou can mask what a gateway logs without changing what it returns, and prove it\nby inspecting the raw payloads on both sides of a single request.
The first working version of _mask used a blocking client ā httpx.Client\ninside an async def. It masked correctly. It also did this:
| run 1 | run 2 | |
|---|---|---|
| no callback | 0.74 s | ā |
blocking httpx.Client | 1.10 s | 0.97 s |
async httpx.AsyncClient | 0.77 s | 0.77 s |
Warm medians, four sequential requests each, first discarded as cold. The async\nfigure repeated exactly; the blocking one did not, which is itself the point ā\nthe penalty depends on what else the loop is doing.
\n![\"Timed](\"https://development-wec.wiline.com/docs/assets/images/gw4-timing-97e67eb7eb08fb9770f75d0b038bb147.png\")
Figure 8. Same request, same masking, one word different in the code.
\nA blocking HTTP call inside an async hook stalls the event loop until Presidio\nanswers, and each request makes several such calls. The masking is not on the\ncaller's critical path ā but it is on everyone else's, because nothing else can\nbe serviced while it waits.
\nChanging httpx.Client to httpx.AsyncClient and awaiting the calls removes\nthe cost: 0.77s against a 0.74s baseline, inside the noise of the model call\nitself, and it reproduced to the hundredth across two separate runs.
These numbers are four sequential requests. Blocking the event loop barely\nshows at one request at a time ā it is under concurrency that the two versions\ndiverge, and we have not measured that here. If you run this at volume, load\ntest it before trusting the table above.
Each request also means two Presidio HTTP calls per message plus two for the\ncompletion. At real volume Presidio becomes a service you size and monitor,\nnot a background detail.
| Logged input | Logged output | Caller's response | Cost | |
|---|---|---|---|---|
filter_scope: input | masked | raw PII | untouched | none |
filter_scope: both | masked | masked | masked | 0.30s guardrail span |
| logging hook | masked | masked | untouched | none measurable |
Worth being clear about the limit of what this achieves: the PII still travels.\nIt reaches the model and it reaches the caller. What has been eliminated is\nretention ā it is no longer sitting in a trace store that a wider group of\npeople can read, months later, long after the request itself is gone.
\nVerified against ghcr.io/berriai/litellm:main-stable, admin UI reporting\nv1.96.2, on 21 August 2026. If a later release adds a logging-scoped output\nmode, prefer it over this.
Check LANGFUSE_HOST is set, not LANGFUSE_BASE_URL, and confirm the variables\nare inside the container with docker exec llm-gateway printenv | grep LANGFUSE.\nA docker compose restart will not have loaded them.
A bad dotted path fails quietly. scrubber.py must sit in the same directory as\nconfig.yaml ā inside the container, not just on the host ā and the object name\nafter the dot must exist. Grep the startup log for ImportError and\nAttributeError.
The deep copy is missing, or the built-in guardrail is still enabled with an\noutput scope. Set default_on: false on it and confirm with\ncurl /guardrails/list.
Check _mask uses httpx.AsyncClient with await, not httpx.Client. See the\nmeasurements above.
The measurements here are sequential, and the failure mode that cost 200-350ms\nis one that only bites properly under concurrency. The next post puts the gateway\nunder parallel load and measures what actually happens to latency when several\nrequests contend for the same event loop.
\n+Part 2 ended with the gateway deciding\nwhich model answers each request ā and still forwarding every prompt verbatim,\nincluding the ones carrying customer names, emails and phone numbers.
\nThis post puts a PII filter in that path. Not in front of the model, though:\nin front of the logs. The model reading\nyour prompt is doing its job; the risk is what gets stored. So the raw text\ngoes to the model and a masked copy goes to your logging.
\nThat is what the gateway advertises. Following its documented configuration got\nme the opposite ā a model that answered correctly and a reply that came back as\n<LOCATION>. This is how to find that and how to fix it.
LiteLLM 1.96.2 (ghcr.io/berriai/litellm:main-stable), Presidio images from\nghcr.io/data-privacy-stack, models qwen-small (Qwen2.5-3B-Instruct) and\nqwen-mid (Qwen3.5-9B) via the WEC Inference API, on 20 August 2026. Defaults\nchange ā if your results differ, check your version first.
Presidio detects personal data\nin text: names, emails, phone numbers, card numbers. You POST it a string and it\nreturns the entities it found, with character offsets and a confidence score.
\nIt ships as two services ā an analyzer that finds PII and an anonymizer\nthat replaces it. The gateway calls both.
\nOne naming note that trips people up: Presidio was Microsoft's project and most\ntutorials still call it that. The repository moved to the data-privacy-stack\norganisation, and github.com/microsoft/presidio now answers Moved Permanently. Same project, new home, which is why the images below aren't under\na Microsoft path.
Presidio only ever talks to the gateway, so it does not need published ports.\nOpen the docker-compose.yml from Part 1 and add two services after the db:\nblock, above volumes::
presidio-analyzer:
image: ghcr.io/data-privacy-stack/presidio-analyzer:latest
container_name: presidio-analyzer
restart: unless-stopped
presidio-anonymizer:
image: ghcr.io/data-privacy-stack/presidio-anonymizer:latest
container_name: presidio-anonymizer
restart: unless-stopped
Two spaces of indentation, same level as litellm: and db:.
Notice what is absent: no ports:. Putting them in the same compose project as\nthe gateway means Docker gives them a shared network and DNS, so the gateway\nreaches them by service name and nothing else on your network can reach them at\nall. If you deployed Presidio separately with ports: [\"5002:3000\"], that\nexposed your PII analyzer to everything that can route to the host ā worth\nundoing.
cd ~/llm-gateway && docker compose up -d
docker compose ps
Four containers, and the Presidio rows should show 3000/tcp with no host\nmapping:
NAME SERVICE STATUS PORTS
llm-gateway litellm Up 127.0.0.1:4000->4000/tcp
llm-gateway-db db Up (healthy) 5432/tcp
presidio-analyzer presidio-analyzer Up (healthy) 3000/tcp
presidio-anonymizer presidio-anonymizer Up (healthy) 3000/tcp
![\"Four](\"https://development-wec.wiline.com/docs/assets/images/gw3-compose-ps-57ff2d738b0cb45457ecc59f8325613d.png\")
If docker compose ps shows only two containers, the compose file was not saved.\nConfirm what compose is actually reading:
docker compose config --services
That must list four names. It reads the file on disk, so it catches an edit that\nnever made it out of your editor.
\nBefore wiring anything, see what Presidio actually finds:
\nThere are no host ports now, so ask from inside the gateway. The LiteLLM image\nships no curl, so use its Python:
docker exec llm-gateway python -c \"import json,urllib.request as u; r=u.Request('http://presidio-analyzer:3000/analyze', data=json.dumps({'text':'Call Rafael on 555-0142 or rafael@example.com','language':'en'}).encode(), headers={'Content-Type':'application/json'}); print(u.urlopen(r).read().decode())\"
[{\"entity_type\": \"EMAIL_ADDRESS\", \"score\": 1.0, \"start\": 27, \"end\": 45},
{\"entity_type\": \"PERSON\", \"score\": 0.85, \"start\": 5, \"end\": 11},
{\"entity_type\": \"URL\", \"score\": 0.5, \"start\": 34, \"end\": 45}]
![\"Analyzer](\"https://development-wec.wiline.com/docs/assets/images/gw3-analyze-scores-16b5f56ac49060ffafbba8dc8a2c2333.png\")
Read that carefully, because it sets expectations for everything after.
\nIt found the email at full confidence and the name at 0.85. It did not find the\nphone number at all. 555-0142 is seven digits with no area code, and\nPresidio validates against real numbering plans rather than pattern-matching\nanything that looks phone-shaped. Add an area code:
docker exec llm-gateway python -c \"import json,urllib.request as u; r=u.Request('http://presidio-analyzer:3000/analyze', data=json.dumps({'text':'Call Rafael on (415) 555-0142 or rafael@example.com','language':'en'}).encode(), headers={'Content-Type':'application/json'}); print(u.urlopen(r).read().decode())\"
Now PHONE_NUMBER appears ā scoring 0.4, against 1.0 for the email.
It is worth checking whether the 555 prefix is the problem, since that range is\nreserved for fiction. It is not. A plausible number behaves identically:
docker exec llm-gateway python -c \"import json,urllib.request as u; r=u.Request('http://presidio-analyzer:3000/analyze', data=json.dumps({'text':'Call Rafael on (415) 682-4531 or on 682-4531','language':'en'}).encode(), headers={'Content-Type':'application/json'}); print(u.urlopen(r).read().decode())\"
[{\"entity_type\": \"PERSON\", \"score\": 0.85, \"start\": 5, \"end\": 11},
{\"entity_type\": \"PHONE_NUMBER\", \"score\": 0.4, \"start\": 15, \"end\": 29}]
Same 0.4 for the full number, and the bare 682-4531 is still missed. So two\nthings are true: an area code is what makes a phone number detectable at all,\nand 0.4 is simply what this recognizer returns for phone numbers ā not a\npenalty for fake ones.
Which means entity types are not equally trustworthy, and confidence is a number\nyou will want to set deliberately. Presidio exposes presidio_score_thresholds\nfor exactly that: a per-entity floor below which detections are discarded. Set it\nat 0.5 and you would silently stop masking every phone number in your traffic.
Do not build a compliance story on the assumption that this catches everything.
\nThere is an escape hatch, and it is worth knowing before you decide the coverage\nis unacceptable. presidio_ad_hoc_recognizers takes a path to a JSON file of\nextra recognizers, loaded when the guardrail starts:
[
{
\"name\": \"LocalPhoneRecognizer\",
\"supported_language\": \"en\",
\"supported_entity\": \"PHONE_NUMBER\",
\"patterns\": [
{ \"name\": \"us-local-7\", \"regex\": \"\\\\b\\\\d{3}-\\\\d{4}\\\\b\", \"score\": 0.9 }
]
}
]
Mount it beside the config and point the guardrail at it:
\n - ./recognizers.json:/app/recognizers.json:ro
presidio_ad_hoc_recognizers: /app/recognizers.json
Recreate the container ā a new volume needs up -d, not restart ā and the same\nprompt that leaked now masks:
Reply OK. Contact <PERSON> on <PHONE_NUMBER> about the invoice.
Why the score is 0.9 and not 0.8. Put 682-4531 inside a sentence ā\nCall me on 682-4531 ā and the default recognizers return DATE_TIME at\n0.85: seven digits with a dash, read as a date. (Pass the number on its own,\nwith no sentence around it, and nothing fires at all; the NER model wants\ncontext before it commits to anything.) At 0.8 your phone recognizer loses the\noverlap and the number is masked as <DATE_TIME>: still redacted, but filed\nunder the wrong entity, which quietly breaks any reporting that counts by type.\nAt 0.86 or above PHONE_NUMBER wins.
Measured across three runs each, entirely deterministic:
\n| Text | Default | Recognizer at 0.8 | Recognizer at 0.9 |
|---|---|---|---|
Call me on 555-0142 | left in the clear | <PHONE_NUMBER> | <PHONE_NUMBER> |
Call me on 682-4531 | <DATE_TIME> | <DATE_TIME> | <PHONE_NUMBER> |
(415) 682-4531 or on 682-4531 | bare one leaks | both masked | both masked |
Note the middle row is not a miss ā it is a misclassification, and the anonymizer\nstill redacts it. The first and third rows are the real leaks, and a recognizer\ncloses both.
\nSeven digits and a dash is a shape, not a meaning, and plenty of things share it.\nRun that recognizer against text from an actual support queue:
\n| Text | Masked as |
|---|---|
Order 482-1099 shipped yesterday | <PHONE_NUMBER> |
Invoice 100-2000 is overdue | <PHONE_NUMBER> |
Part number 250-4000 is discontinued | <PHONE_NUMBER> |
The error code was 500-1001 | <PHONE_NUMBER> |
Our office is at 200-4500 Main Street | <PHONE_NUMBER> |
RFC 793-1981 defines TCP | <PHONE_NUMBER> |
Six for six. And the 0.9 you set to win the DATE_TIME overlap now wins every\noverlap, so the recognizer does not merely add noise ā it overwrites correct\nclassifications with a wrong one. Traces full of <PHONE_NUMBER> where the order\nnumbers used to be are worse than traces with one number in the clear, because\nnow you cannot tell which is which.
The fix is to stop matching on shape alone and require a cue word in front of the\ndigits. That has to be a lookbehind: Presidio redacts the whole match, so\n(call\\W+)(\\d{3}-\\d{4}) would swallow the word \"call\" along with the number ā\ncapture groups do not narrow the span. Python's re only allows fixed-width\nlookbehinds, which is why this is an alternation rather than one tidy list:
[
{
\"name\": \"LocalPhoneRecognizer\",
\"supported_language\": \"en\",
\"supported_entity\": \"PHONE_NUMBER\",
\"patterns\": [
{
\"name\": \"us-local-7-cued\",
\"regex\": \"(?i)(?:(?<=call )|(?<=called )|(?<=calling )|(?<=phone )|(?<=phone: )|(?<=phone is )|(?<=tel )|(?<=tel: )|(?<=cell )|(?<=cell: )|(?<=cell is )|(?<=mobile )|(?<=mobile: )|(?<=contact )|(?<=contact: )|(?<=number )|(?<=number: )|(?<=number is )|(?<=me on )|(?<=him on )|(?<=her on )|(?<=them on )|(?<=us on )|(?<=me at )|(?<=him at )|(?<=her at )|(?<=them at )|(?<=us at ))(?<!part number )(?<!serial number )(?<!order number )(?<!invoice number )(?<!account number )(?<!model number )(?<!tracking number )(?<!reference number )(?<!batch number )(?<!ticket number )(?<!case number )\\\\d{3}-\\\\d{4}\\\\b\",
\"score\": 0.9
}
]
}
]
The trailing negative lookbehinds are there because number is a useful cue and\npart number is not.
Measured over twelve phone phrasings and twelve lookalikes:
\n\\b\\d{3}-\\d{4}\\b | Cue-anchored | |
|---|---|---|
| Phone numbers masked | 12/12 | 12/12 |
| Lookalikes wrongly masked | 12/12 | 0/12 |
Two things this does not fix, and you should know both. Order 482-1099 still\ncomes back as <DATE_TIME> ā that is spaCy misfiring on its own, with or without\nyour recognizer. And a phone number introduced by a phrase you did not think of\ngoes back to leaking. A cue list is a guess about how people write, so treat it\nas something to revisit against your own traffic rather than a finished artefact.
BR is in the default region list, so (11) 91234-5678 is recognised ā and\n91234-5678 without the DDD is not, exactly as with a missing US area code. The\nsame recognizer closes it with \\d{4,5}-\\d{4} and a Portuguese cue list.
The gateway calls Presidio through a guardrail. Open config.yaml and add a\nguardrails: block at the end ā this is the configuration the LiteLLM\ndocumentation gives for masking only on the logging path:
guardrails:
- guardrail_name: presidio-log-mask
litellm_params:
guardrail: presidio
mode: logging_only
default_on: true
presidio_analyzer_api_base: http://presidio-analyzer:3000
presidio_anonymizer_api_base: http://presidio-anonymizer:3000
mode: logging_only is the important one. The documentation describes it as:
\n\nRun after LLM call, only apply PII Masking before logging to Langfuse, etc.\nNot on the actual llm api request / response.
\n
default_on: true applies it to every request rather than only those that ask\nfor it by name. And the two api_base values are the service names from the\ncompose file ā the guardrail runs inside the gateway container, so localhost\nwould point at the wrong place.
Restart, and wait for the proxy to actually come back ā it takes 30 to 60\nseconds, which is longer than most sleep commands people put in front of it:
docker compose restart litellm
until curl -sf localhost:4000/health/readiness >/dev/null 2>&1; do printf '.'; sleep 3; done; echo \" ready\"
Then confirm the guardrail loaded, which is more reliable than reading logs:
\nKEY=$(grep -m1 LITELLM_MASTER_KEY ~/llm-gateway/.env | cut -d= -f2- | tr -d '\"') && \\
curl -s localhost:4000/guardrails/list -H \"Authorization: Bearer $KEY\" \\
| python3 -c \"import sys,json; g=json.load(sys.stdin)['guardrails'][0]; p=g['litellm_params']; print(json.dumps({'guardrail_name': g['guardrail_name'], **{k: p.get(k) for k in ('guardrail','mode','presidio_filter_scope','default_on','presidio_analyzer_api_base','presidio_anonymizer_api_base','fail_on_error','unreachable_fallback')}}, indent=2))\"
The endpoint returns every possible guardrail field, most of them null, so this\npicks out the ones that matter:
{
\"guardrail_name\": \"presidio-log-mask\",
\"guardrail\": \"presidio\",
\"mode\": \"logging_only\",
\"presidio_filter_scope\": \"input\",
\"default_on\": true,
\"presidio_analyzer_api_base\": \"http://presidio-analyzer:3000\",
\"presidio_anonymizer_api_base\": \"http://presidio-anonymizer:3000\",
\"fail_on_error\": true,
\"unreachable_fallback\": \"fail_closed\"
}
Those last two are the defaults worth knowing before this sees real traffic: if\nPresidio is unreachable, requests fail rather than passing unmasked.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw3-guardrail-list-cd2a45b09b50cc051970b5b85feba496.png\")
Now the part that matters. Send a prompt whose answer proves whether the model\nsaw the real text ā asking it to echo the PII back is useless, because the reply\ngets scanned too:
\nKEY=$(grep -m1 LITELLM_MASTER_KEY ~/llm-gateway/.env | cut -d= -f2- | tr -d '\"') && \\
curl -s localhost:4000/v1/chat/completions \\
-H \"Authorization: Bearer $KEY\" -H 'Content-Type: application/json' \\
-d '{\"model\":\"qwen-small\",\"temperature\":0,\"messages\":[{\"role\":\"user\",\"content\":\"Which US city does this area code belong to: (415) 555-0142? Answer with only the city name.\"}]}' \\
| python3 -c \"import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'])\"
Expected: San Francisco, proving the model read the digits.
Actual:
\n<LOCATION>
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw3-masked-answer-1467b181d83013e63ad42b11ffcdfa95.png\")
The model answered a question that requires reading a real area code ā and then\nthe answer itself came back masked. logging_only was supposed to stay off the\nactual response.
There is a second setting, and its default is doing the work.
\npresidio_filter_scope controls which direction gets scanned. From the\ndocumentation:
\n\n\n
input: only user ā model content is scanned;output: only model ā user\ncontent is scanned;both(default): scan both directions
And, on the same page:
\n\n\nUse
\npresidio_filter_scope: output(orboth) when you want Presidio to\nactively scan and mask the model's response before it reaches the user.
So both ā the default ā actively masks responses. The documented logging_only\nexample does not set presidio_filter_scope at all, which leaves it at both.\nThe result is a configuration that masks the thing the same page says it will not\ntouch.
In the source, initialize_presidio reads that scope and registers a second\ncallback with apply_to_output=True on post_call whenever output scanning is\nenabled. That callback's event hook is hard-coded, so the mode you set never\nreaches it.
This has been reported.\nIssue #30447, \"logging_only\nPresidio guardrail corrupts user-facing response (assistant content replaced with\nPII tokens)\", was filed on 15 June 2026 and closed the next day with no\ndiscussion, alongside a pull request titled \"fix(presidio): don't mask the live\nrequest when guardrail is logging_only\". The report was about the response; the\nfix addressed the request. On the main-stable image used here, the behaviour\nabove still reproduces.
A second report is open:\n#35951 notes that in\nlogging_only mode the masking hook rewrites kwargs[\"messages\"] but does not\npropagate to standard_logging_object, which is what downstream loggers read.
One line. Add presidio_filter_scope: input to the guardrail:
guardrails:
- guardrail_name: presidio-log-mask
litellm_params:
guardrail: presidio
mode: logging_only
presidio_filter_scope: input
default_on: true
presidio_analyzer_api_base: http://presidio-analyzer:3000
presidio_anonymizer_api_base: http://presidio-anonymizer:3000
Restart, wait for readiness, and run the same request:
\nSan Francisco
You can run Presidio beside a gateway, mask prompts at logging time without\ntouching the live request, and tell the two directions apart with\npresidio_filter_scope.
One right answer could be a lucky guess ā \"San Francisco\" is a plausible default\nfor a masked area-code question, and temperature: 0 makes a guess repeat just\nas reliably as a real answer. So ask about several area codes the model cannot\nbluff:
KEY=$(grep -m1 LITELLM_MASTER_KEY ~/llm-gateway/.env | cut -d= -f2- | tr -d '\"')
for ac in 907 808 216 505; do
printf \"area %s -> \" \"$ac\"
curl -s localhost:4000/v1/chat/completions \\
-H \"Authorization: Bearer $KEY\" -H 'Content-Type: application/json' \\
-d \"{\\\"model\\\":\\\"qwen-mid\\\",\\\"temperature\\\":0,\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"Which US state does this phone number's area code belong to: ($ac) 555-0142? Answer with only the state name.\\\"}]}\" \\
| python3 -c \"import sys,json; print(' '.join(json.load(sys.stdin)['choices'][0]['message']['content'].split()))\"
done
area 907 -> Alaska
area 808 -> Hawaii
area 216 -> Ohio
area 505 -> New Mexico
![\"Four](\"https://development-wec.wiline.com/docs/assets/images/gw3-area-codes-9448b7ab1d15d52e9cafb0d8ad0e5b86.png\")
Four for four. The model is reading the digits, so the prompt is reaching it\nunmasked. That is the behaviour this post set out to get.
\nThe whitespace collapse in that command is not cosmetic fussiness ā qwen-mid is\na reasoning model and prefixes its answers with newlines, inconsistently enough\nthat .strip() alone leaves ragged output.
It is the obvious question, and the obvious answer is wrong. mode: pre_call\nmasks the prompt before the model ever sees it ā which sounds like the safest\npossible arrangement.
Change the one line and find out:
\n mode: pre_call
Restart, then ask the same question:
\nKEY=$(grep -m1 LITELLM_MASTER_KEY ~/llm-gateway/.env | cut -d= -f2- | tr -d '\"') && \\
curl -s localhost:4000/v1/chat/completions \\
-H \"Authorization: Bearer $KEY\" -H 'Content-Type: application/json' \\
-d '{\"model\":\"qwen-small\",\"temperature\":0,\"max_tokens\":60,\"messages\":[{\"role\":\"user\",\"content\":\"Which US state does this phone number'\"'\"'s area code belong to: (907) 555-0142? Answer with only the state name.\"}]}' \\
| python3 -c \"import sys,json; print(' '.join(json.load(sys.stdin)['choices'][0]['message']['content'].split()))\"
To determine the state for a given phone number's area code, I would need to know
the specific area code of the phone number provided. Please provide the area code
or the full phone number so I can identify the corresponding state.
The exact wording moves around between runs even at temperature: 0 ā the point\nis that every version of it asks you for the number you already sent.
![\"Under](\"https://development-wec.wiline.com/docs/assets/images/gw3-precall-blind-12b8776adeda133b79cac28a0bb36166.png\")
The model is politely asking for the number you just gave it. It received\n<PHONE_NUMBER> and there is nothing in a placeholder to reason about. Nothing\nfailed, nothing errored ā you simply got a useless answer, which is the failure\nmode that is hardest to notice in production.
qwen-mid behaved worse in my runs: the same request timed out twice at 40\nseconds with max_tokens: 60, where it answered in well under a second on\nunmasked input. I would not claim a mechanism from two samples, but if you are\nrouting to a reasoning model, time this before trusting it.
So the three placements, same question each time:
\n| Configuration | Model receives | Result |
|---|---|---|
mode: pre_call | <PHONE_NUMBER> | asks you for the number it was given |
mode: logging_only, scope left default | real digits | correct answer, returned as <LOCATION> |
mode: logging_only, presidio_filter_scope: input | real digits | San Francisco |
Only the third gives you a working application. Put mode back to\nlogging_only before continuing.
You now have prompts reaching models intact, with PII masking attached to the\nlogging path instead of the request path ā and one setting away from silently\nredacting your users' answers.
\nOne thing this fix does not cover, and it matters: input scope masks the\nprompt only. Whatever the model says is logged verbatim ā and a model asked\nabout a customer will repeat that customer's name back. So the PII you removed\nfrom the prompt reappears in the completion. Widening the scope to both closes\nthat hole and reintroduces the response corruption above; there is no setting\nthat does both.\nPart 4 measures the leak\nand builds the missing piece on async_logging_hook, the documented callback\nthat runs after the response has been sent.
Being straight about the boundary: the masked copy lands wherever your logging\ncallbacks send it. The gateway's own LiteLLM_SpendLogs table is not one of\nthose ā messages stayed {} in every configuration I tried, including with\nstore_prompts_in_spend_logs set both in config.yaml and as an environment\nvariable. The documentation points at Langfuse and similar destinations, which is\nwhere the next post looks. Until you have wired one up and seen a masked\npayload in it, treat the masking half as unproven in your own deployment.
Two more things worth knowing before this sits in front of real traffic. The\nguardrail defaults to fail_on_error: true with unreachable_fallback: fail_closed, so if Presidio goes down your requests fail rather than passing\nunmasked ā the right default, but plan for it. And every Presidio call adds\nlatency to the logging path; Part 2's timing method applies here too.
docker compose ps shows two containers, not fourāThe compose file was not saved. docker compose config --services reads what is\non disk ā if Presidio is missing there, the edit did not land.
curl: executable file not found in $PATHāThe LiteLLM image has no curl. Probe with the python -c one-liner above.
Name or service not known reaching presidio-analyzerāThe two stacks are on different Docker networks. Both sets of services must be in\nthe same compose project, or share an external network.
\nValidationError: mode Field requiredāmode is mandatory on a guardrail. Omitting it exits the proxy on startup ā\nApplication startup failed. Exiting. You cannot express logging-only by leaving\nmode out.
<LOCATION> or <PHONE_NUMBER>āpresidio_filter_scope is at its both default. Set it to input.
Check the score. 555-0142 is not detected at all; (415) 555-0142 scores 0.4.\nUse presidio_score_thresholds to set a per-entity floor.
config.yaml seem to have no effectāEdit the file, then restart ā in that order. And a cd from an earlier step\ncarries over, so check which directory you are in before running docker compose.
.env changes are ignoredādocker compose restart does not reload environment. Use\ndocker compose up -d --force-recreate litellm.
Masking is configured, but you have not yet watched a masked payload land\nanywhere. The next post wires a real logging destination to the gateway and looks\nfor <PERSON> in a trace ā closing the loop this one deliberately leaves open ā\nand then turns to budgets and what the whole arrangement actually costs.
presidio_filter_scope and presidio_score_thresholds+![\"Qwen\"](\"https://development-wec.wiline.com/docs/assets/images/qwen-209001e8f781e3651b8d7b863343f2dc.png\")
+Part 1 ended on an uncomfortable\nnumber. The same three-word answer cost 3 tokens from a small model and 200 from\na reasoning model ā which spent all 200 thinking and returned nothing at all.
\nEvery request your apps send picks a model, and mostly that choice is made once,\nhardcoded, and never revisited. This post puts the gateway in charge of it\ninstead: classify the request, route it to a model sized for the work. Then it\nmeasures what that decision costs, because it is not free and most write-ups\nskip that part.
\n\n\nThe gateway from Part 1 already has three models registered. A router is just\nanother entry in model_list that maps complexity tiers onto them:
- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: qwen-small
MEDIUM: qwen-mid
COMPLEX: qwen-large
REASONING: qwen-large
return_raw_model_name: true
return_raw_model_name: true is the important one for now. Without it the\nresponse reports smart-router and you have no idea which model served you.\nWith it, the response names the model that actually ran ā so every test below\nis self-verifying.
docker compose restart litellm
You'll send the same request many times with only the prompt changing, so wrap\nit once:
\nask() { curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d \"{\\\"model\\\":\\\"smart-router\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"$1\\\"}],\\\"max_tokens\\\":10}\" | jq -r .model; }
Then five prompts of increasing difficulty:
\nask \"hi\"
ask \"what is a vpc\"
ask \"write a python function that retries an http call with backoff\"
ask \"our two services deadlock under load, walk me through diagnosing it\"
ask \"prove that the halting problem is undecidable\"
One line per prompt, in the order sent:
\nQwen2.5-3B-Instruct
Qwen2.5-3B-Instruct
Qwen3.5-9B
Qwen2.5-3B-Instruct
Qwen2.5-3B-Instruct
![\"Five](\"https://development-wec.wiline.com/docs/assets/images/gw2-heuristic-route-af0d1b1a3a318287720fa1b18160e760.png\")
\nFigure 1. The code request went up a tier. The two hardest questions did not.
A greeting on the small model is right. A Python request on the mid model is\nright. But a distributed-systems deadlock and one of the hardest questions in\ncomputer science both landed on a 3-billion-parameter model.
\nTo understand why, you have to know what the router is actually doing.
\nBy default this router makes zero API calls. It scores the prompt locally with\npattern matching ā that's what \"heuristic\" means here: a cheap rule of thumb\nthat approximates a judgement without making it.
\nIt scores seven dimensions, each producing a value between ā1 and +1, then\nmultiplies each by a fixed weight and adds them up:
\n| Dimension | Weight | Fires on |
|---|---|---|
codePresence | 0.30 | function, class, api, schema, ā¦ |
reasoningMarkers | 0.25 | \"step by step\", \"think through\", \"analyze\" |
technicalTerms | 0.25 | \"architecture\", \"distributed\", \"encryption\" |
tokenCount | 0.10 | ā1.0 under 15 tokens, +1.0 over 400 |
simpleIndicators | 0.05 | \"what is\", \"define\", greetings ā scores ā1.0 |
multiStepPatterns | 0.03 | \"firstā¦ then\", numbered steps |
questionComplexity | 0.02 | more than three question marks |
The weighted sum maps to a tier at three boundaries: below 0.15 is SIMPLE,\nbelow 0.35 MEDIUM, below 0.60 COMPLEX, and above that REASONING. Those\nvalues are all documented and all configurable.
\nTwo dimensions can push the score down. A short prompt scores ā1.0 on\ntokenCount. A prompt containing \"what is\" scores ā1.0 on simpleIndicators.
The router records its own working. Every routed request writes a decision into\nthe spend log:
\ncurl -s http://127.0.0.1:4000/spend/logs \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
| jq -r '.[].metadata.routing_decision | select(.) | \"\\(.tier) \\(.score) -> \\(.routed_model) \\(.signals)\"' | head -5
Newest first, so this reads bottom-up against the order the prompts were sent:
\nSIMPLE -0.1 -> qwen-small [\"short (11 tokens)\"]
SIMPLE 0 -> qwen-small null
MEDIUM 0.3 -> qwen-mid [\"code (function, python)\"]
SIMPLE -0.15000000000000002 -> qwen-small [\"short (3 tokens)\",\"simple (what is)\"]
SIMPLE -0.15000000000000002 -> qwen-small [\"short (0 tokens)\",\"simple (hi)\"]
![\"Routing](\"https://development-wec.wiline.com/docs/assets/images/gw2-decisions-c9f72192a937dfe8928c7a438f555144.png\")
\nFigure 2. Not a black box ā the router reports the score and the signals it fired.
Take the last one. prove that the halting problem is undecidable:
tokenCount -1.0 Ć 0.10 = -0.10 11 tokens, under the 15 threshold
codePresence 0.0 Ć 0.30 = 0.00 no code keywords
reasoningMarkers 0.0 Ć 0.25 = 0.00 \"prove\" is not in the marker list
technicalTerms 0.0 Ć 0.25 = 0.00
simpleIndicators 0.0 Ć 0.05 = 0.00
multiStepPatterns 0.0 Ć 0.03 = 0.00
questionComplexity 0.0 Ć 0.02 = 0.00
------
-0.10 below 0.15 ā SIMPLE ā 3B model
Every term is zero except a penalty for being short. The question gets routed\ndown because it is brief.
\nNow the deadlock prompt, which is worse: it scored 0.00 with no signals at\nall. Not a low score ā nothing matched. \"Our two services deadlock under load,\nwalk me through diagnosing it\" contains no keyword the scorer recognises, so it\nfalls to SIMPLE by default.
\nYou can read a routing decision ā tier, score, and the signals that produced it\nā and reproduce the arithmetic by hand from the dimension weights.
The router isn't malfunctioning. It is doing exactly what its rules say. The\nrules just have no way to see difficulty that isn't spelled out in vocabulary\nit knows.
\nThe failure mode is systematic, not random: short prompts that need deep\nthinking get routed down. Those are also the prompts where a wrong model is\nmost obvious to the user.
Single-word keywords are matched on word boundaries, so endpoint does not match\nendpoints. Every single-word keyword in the default lists is singular, and none\nof them match a plural.
Whether that changes anything depends on how close the score already sits to a\nboundary, because the dimensions are stepped rather than linear ā technicalTerms\nscores 0.5 at two matches and 1.0 at four, so losing one match often changes\nnothing. Sometimes it changes the model:
Review our api endpoint for authentication and authorization problems ā COMPLEX +0.425
Review our api endpoints for authentication and authorization problems ā MEDIUM +0.275
One letter, one tier. The direction is always the same: a plural scores lower or\nequal, never higher, so the drift is toward the cheaper model.
\nPlurals you care about can be appended to the technical list with\ncustom_technical_keywords. There is no equivalent for code keywords ā the only\nlever is code_keywords, which replaces the built-in list rather than extending\nit.
The alternative is to spend a model call on the decision. Four lines:
\n classifier_type: llm
classifier_llm_config:
model: qwen-small
timeout_ms: 3000
Now the router sends the prompt to qwen-small with a rubric and a schema that\nforces back exactly one of SIMPLE, MEDIUM, COMPLEX, REASONING, and routes\non the answer. The classifier here is the cheapest model we have ā the same 3B\nthat was wrongly answering the hard questions a moment ago. It turns out to be a\nbetter judge of difficulty than it is an answerer of it.
Restart, then re-run the identical five prompts:
\nQwen2.5-3B-Instruct
Qwen3.5-9B
Qwen3.5-122B
Qwen3.5-122B
Qwen3.5-122B
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw2-llm-route-9a7f2aa9f50bca506a1786ae36cf8c3b.png\")
\nFigure 3. Both failures corrected ā and two prompts moved up that arguably shouldn't have.
| Prompt | Heuristic | LLM classifier |
|---|---|---|
hi | 3B | 3B |
what is a vpc | 3B | 9B |
python function ā¦ backoff | 9B | 122B |
deadlock under load | 3B ā | 122B ā |
halting problem | 3B ā | 122B ā |
The two broken cases are fixed. But read rows 2 and 3 again ā everything moved\nup. \"What is a vpc\" is a factual lookup a 3B model answers perfectly well, and\nit now runs on the 9B. You stopped under-serving hard prompts and started\nover-serving easy ones. Whether that trade is worth it depends on your traffic\nmix, and you should measure yours rather than trust this table.
\nThis is the part that gets left out. Add a twin that skips the router, then time\nboth:
\ndirect() { curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d \"{\\\"model\\\":\\\"qwen-small\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"$1\\\"}],\\\"max_tokens\\\":10}\" | jq -r .model; }
time ask \"hi\"
time ask \"hi\"
time direct \"hi\"
Qwen2.5-3B-Instruct real 0m2.474s ā first call after restart
Qwen2.5-3B-Instruct real 0m1.041s
qwen-small real 0m0.529s
![\"time](\"https://development-wec.wiline.com/docs/assets/images/gw2-timing-ea97fdf0902b60a49ea16f3951a173c6.png\")
\nFigure 4. Same prompt, same model answers it, twice the wall clock.
The same 3B model produced both answers. The only difference is that one request\nwas classified first. Now decompose it ā the spend log times each call\nseparately:
\n| Wall clock | Classifier call | Serving call |
|---|---|---|
| 2.474 s (cold) | 1449 ms | 806 ms |
| 1.041 s | 549 ms | 456 ms |
| 0.529 s (direct) | ā | 495 ms |
The arithmetic closes: 549 + 456 = 1005 ms against 1.041 s measured at the shell.
\nThree things fall out:
\nThe overhead is not a constant. Across the day's samples the classifier call\nranged from 476 ms to 1449 ms, and the first request after a restart cost 1449 ms\non its own. It is a full inference call and inherits whatever the backend is\ndoing. Any single number you quote for it is the number you happened to catch.
\nToken cost is fixed and larger than you'd guess. Each classification sent\n281ā293 input tokens ā the rubric ā and returned about 10. Routing hi, a\none-token prompt, costs ~281 tokens of classification before anything answers.
The serving call slows down too. 456ā806 ms when routed, against 495 ms\ndirect for the identical request. The classifier appears to leave the backend\nbusy for the request queued behind it. You would never see this from wall clock\nalone.
\nPaired wall-clock timing tells you that it got slower, never where. Decompose\ninto the classifier call and the serving call before you quote a figure ā the\nfirst honest number here was almost double the one a single time run suggested.
If the classification call times out, returns the wrong shape, or comes back\nempty, the router falls back to the heuristic scorer ā the thing whose\nfailures you just paid half a second to avoid. Silently.
\nThe official docs name the way out: set classifier_fallback: default_model and\na timeout routes to a model you chose deliberately, rather than to the scorer\nthat sends hard questions to a 3B model.
Also worth checking rather than assuming: the docs give timeout_ms a default of\n2000, while the package in our container defaults to 3000. Read the version you\nactually installed.
| Heuristic | LLM classifier | |
|---|---|---|
| Latency added | sub-millisecond | 476ā1449 ms |
| Extra API calls | none | one per request |
| Extra input tokens | none | ~280 per request |
| Hard-but-short prompts | routed down | routed correctly |
| Easy prompts | routed correctly | routed up |
| Fails by | being confidently wrong | timing out, then being confidently wrong |
The heuristic is the right default for high-volume traffic that looks alike, and\nyou can improve it a lot with custom_technical_keywords for your own domain\nvocabulary. The LLM classifier earns its cost when prompts are varied, when\ngetting the model wrong is expensive, and when half a second doesn't matter ā\nbatch work, agents, anything already taking seconds.
What neither of them is, is free.
\nsmart-router instead of a model nameāreturn_raw_model_name isn't set. Without it the router echoes the alias you\nasked for and you can't tell what served the request.
Check the signals field in the routing decision. If it's null, no dimension\nmatched at all and the prompt scored 0.00, which lands in SIMPLE. That's a sign\nyour traffic doesn't use the vocabulary the default keyword lists expect.
They're on the routed request, under metadata.routing_decision, not on the\nclassifier call. Filter with map(select(.)) as above ā half the rows are the\nclassifier calls themselves and carry no decision.
Cold start. The first classification here took 1449 ms against roughly 550 ms\nonce warm. Discard the first sample when timing.
\nThe gateway now decides which model runs, and you know what that decision costs\nin latency and tokens. It still forwards every prompt verbatim to whichever model\nwins ā including the ones containing customer names, emails and API keys.
\nThe next post puts a filter in that path.
\n+Here's how it usually goes. One app needs a model, so you paste the API key into\nits .env. Then a second app needs one. Then a script. Six months later the same\nkey is in five places, nobody remembers which of them is still running, and you\ncan't rotate it without breaking something you'll only find out about when it\nbreaks.
A gateway is the boring fix. One endpoint in front of every model, one place that\nholds the real credential, and a scoped key per app that you can revoke on its own.\nThis post deploys one on a WEC Instance and points it at the WEC Inference API.
\n\n\nThe host already runs five other stacks, which matters ā this is the normal case,\nnot a clean VM. Before adding anything, check what's listening and what's free:
\ndocker ps --format '{{.Names}}\\t{{.Ports}}'
sudo ss -tlnp | grep -E ':(4000|5432|5433)\\b'
langfuse-postgres-1 127.0.0.1:5433->5432/tcp
langfuse-clickhouse-1 127.0.0.1:8123->8123/tcp, 127.0.0.1:9000->9000/tcp
openclaw-caddy-1 100.87.239.229:80->80/tcp, 100.87.239.229:443->443/tcp
...
LISTEN 0 244 127.0.0.1:5432 users:((\"postgres\",pid=892))
LISTEN 0 4096 127.0.0.1:5433 users:((\"docker-proxy\"))
Port 4000 is free. Postgres 5432 belongs to the host and 5433 to Langfuse, so the\ngateway's database gets neither ā it won't publish a port at all.
\nThen confirm the backend answers before putting anything in front of it:
\ncurl -s https://inference.wiline.com/v1/models \\
-H \"Authorization: Bearer $WEC_API_KEY\" | jq -r '.data[].id'
whisper-large-v3
gemma4
wiline-coding
zai-org/GLM-5.2
Qwen2.5-3B-Instruct
Qwen3.5:9B
Qwen3.5-122B
kokoro
Llama3.1-8B-Instruct
bge-m3
Qwen3.5-9B
wiline-auto
wiline-cost
Thirteen models on one key. That's the thing we're about to stop handing out.
\nTwo containers: the gateway and a Postgres for its own state. Make a directory and\nwrite the compose file:
\nmkdir -p ~/llm-gateway && cd ~/llm-gateway
services:
litellm:
image: ghcr.io/berriai/litellm:main-stable
container_name: llm-gateway
restart: unless-stopped
ports:
- \"127.0.0.1:4000:4000\"
volumes:
- ./config.yaml:/app/config.yaml:ro
command: [\"--config\", \"/app/config.yaml\", \"--port\", \"4000\"]
env_file: .env
depends_on:
db:
condition: service_healthy
db:
image: postgres:17
container_name: llm-gateway-db
restart: unless-stopped
user: \"999:999\"
environment:
POSTGRES_USER: llmproxy
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: litellm
volumes:
- gateway-db:/var/lib/postgresql/data
healthcheck:
test: [\"CMD-SHELL\", \"pg_isready -U llmproxy -d litellm\"]
interval: 5s
timeout: 5s
retries: 10
volumes:
gateway-db:
Four of those lines are doing real work:
\n127.0.0.1:4000:4000 keeps the gateway off the public internet. It holds every\nmodel credential you own, so publishing it to 0.0.0.0 would put all of them\nbehind a single shared password. Part 1 of the hardening\nseries is the long version of why\nthat's a bad trade.
The database has no ports: at all. Nothing outside the compose network needs to\nreach it, and 5432 was taken anyway.
user: \"999:999\" runs Postgres as its built-in non-root user from the first boot.\nPart 2 covers what that buys you\nand how to find the right UID for an image instead of guessing.
condition: service_healthy matters more than it looks. The gateway runs database\nmigrations on startup ā without it, it races Postgres and crash-loops.
model_list:
- model_name: qwen-small
litellm_params:
model: openai/Qwen2.5-3B-Instruct
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
- model_name: qwen-mid
litellm_params:
model: openai/Qwen3.5-9B
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
- model_name: qwen-large
litellm_params:
model: openai/Qwen3.5-122B
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
- model_name: embeddings
litellm_params:
model: openai/bge-m3
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
store_model_in_db: true
litellm_settings:
drop_params: true
model_name is the alias your apps ask for. model: is what the gateway actually\ncalls. That indirection is most of the value ā swap qwen-mid to point somewhere\nelse next month and not one client changes.
The openai/ prefix says \"speak the OpenAI protocol to a custom api_base.\" The\nWEC Inference API is OpenAI-compatible, so that's all it takes.
Generate the passwords rather than inventing them:
\nPGPW=$(openssl rand -hex 16)
MK=\"sk-$(openssl rand -hex 20)\"
cat > .env <<EOF
POSTGRES_PASSWORD=${PGPW}
DATABASE_URL=postgresql://llmproxy:${PGPW}@db:5432/litellm
LITELLM_MASTER_KEY=${MK}
WEC_API_KEY=your-wec-key-here
EOF
chmod 600 .env
Then put your real WEC key in place of your-wec-key-here. The master key is the\ngateway's root credential ā it can create keys, delete keys and reach every model,\nso it goes in your password manager and nowhere else.
docker compose up -d
ā Image ghcr.io/berriai/litellm:main-stable Pulled 58.2s
ā Network llm-gateway_default Created 0.1s
ā Volume llm-gateway_gateway-db Created 0.1s
ā Container llm-gateway-db Healthy 8.8s
ā Container llm-gateway Created 0.2s
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/gw-compose-up-e741e826f043d86aa7f1a7214f71fbcb.png\")
\nFigure 1. One pull, two containers, a network and a volume.
Created rather than Started on that last line is normal for the compose output,\nbut check anyway ā and check what the image cost you in disk while you're there:
docker compose ps -a && echo && df -h /
NAME IMAGE SERVICE STATUS PORTS
llm-gateway ghcr.io/berriai/litellm:main-stable litellm Up 3 minutes 127.0.0.1:4000->4000/tcp
llm-gateway-db postgres:17 db Up 3 minutes (healthy) 5432/tcp
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 58G 54G 4.2G 93% /
Note the database's PORTS column ā 5432/tcp with no host binding in front of\nit. The gateway image is 1.16GB, which on a box already running five stacks is not\nnothing. Check you have the room before you start, not after.
Two lines in the startup log are worth reading, because both look worse than they are:
\ndocker compose logs litellm --tail 40
ā
Migration diff applied successfully
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
register_model: model=openai/Qwen2.5-3B-Instruct not in built-in cost map and no
prefix/region variant matched; cache cost fields will default to 0.
![\"Container](\"https://development-wec.wiline.com/docs/assets/images/gw-ps-logs-ed50299979281a4fb3dfd6ec9c89d189.png\")
\nFigure 2. Migrations applied, application startup complete, and one cost-map\nwarning per model.
0.0.0.0:4000 is the bind inside the container. The host publish is still\n127.0.0.1, which is what actually gates access. And the cost-map warning is\nnarrower than it reads ā it's about cache pricing specifically. Regular token\npricing still happens, which turns out to matter later.
set -a && . ./.env && set +a
curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: gateway works\"}],\"max_tokens\":20}' | jq
\"finish_reason\": \"length\",
\"message\": {
\"reasoning_content\": \"Thinking Process:\\n\\n1. **Analyze the Request:**\\n * Input: \\\"\",
\"content\": null
}
\"usage\": {\"completion_tokens\": 20, \"prompt_tokens\": 16, \"total_tokens\": 36}
![\"Raw](\"https://development-wec.wiline.com/docs/assets/images/gw-reasoning-null-2d60fce2efcb8e175da27120082ac585.png\")
\nFigure 3. A successful call with nothing in it.
content: null. The call worked ā it routed, it hit the backend, it counted tokens\nā but there's no answer in it.
Qwen3.5-9B is a reasoning model. It spent all twenty tokens thinking and had none\nleft to speak with. The obvious move is to give it more room:
curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: gateway works\"}],\"max_tokens\":200}' | jq '.choices[0].message.content, .usage'
null
{
\"completion_tokens\": 200,
\"prompt_tokens\": 16,
\"total_tokens\": 216
}
Two hundred tokens, still nothing. And this is the part worth pausing on: an earlier\nrun of that exact command did answer, at 192 tokens. Same model, same prompt,\ndifferent amount of thinking. There's no max_tokens you can set that reliably\nbuys you an answer, because the reasoning length isn't fixed.
Now the same prompt through the small model, at the original twenty tokens:
\ncurl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: gateway works\"}],\"max_tokens\":20}' | jq '.choices[0].message.content, .usage'
\"gateway works\"
{
\"completion_tokens\": 3,
\"prompt_tokens\": 35,
\"total_tokens\": 38
}
![\"Reasoning](\"https://development-wec.wiline.com/docs/assets/images/gw-token-contrast-922bab5aa94a6417e02dbc0d6fe0c874.png\")
\nFigure 4. Two hundred tokens and no answer, against three tokens and the answer.
Nothing is broken here. The reasoning model is doing exactly what it's built to do,\non a question that didn't need it. The fix isn't a bigger token budget ā it's not\nsending trivial work to that model in the first place. Which is the entire argument\nfor routing, and the subject of the next post.
\nYou can put a gateway in front of several models, give each one an alias your apps\ncall instead of a provider model ID, and read a response well enough to tell a\nreasoning model from a plain one.
A gateway that holds every credential you own should refuse anyone without a key.\nWorth checking rather than assuming:
\ncurl -s -o /dev/null -w \"no key: %{http_code}\\n\" http://127.0.0.1:4000/v1/chat/completions \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'
curl -s -o /dev/null -w \"wrong key: %{http_code}\\n\" http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer sk-not-a-real-key\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'
curl -s -o /dev/null -w \"master key: %{http_code}\\n\" http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":5}'
no key: 401
wrong key: 401
master key: 200
![\"Three](\"https://development-wec.wiline.com/docs/assets/images/gw-auth-401-a9d9d89ff8432a6ec7a29486800ea82c.png\")
\nFigure 5. Anonymous and forged both rejected, the real key through.
The gateway ships a web interface, and it's bound to localhost ā which is the\npoint, so reach it over SSH rather than opening a port:
\nssh -L 4000:127.0.0.1:4000 ubuntu@your-wec-instance
Leave that running and open http://localhost:4000/ui. Username admin, password\nis the master key from your .env.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw-ui-login-94d370f9590bd022684fb80ccabb2aff.png\")
\nFigure 6. The login page states its own default credentials, which is a good\nreminder that the master key is the only thing standing there.
Click Models + Endpoints:
\n![\"Model](\"https://development-wec.wiline.com/docs/assets/images/gw-ui-models-4372c39b1bfb45a2d61782e145f1c5e2.png\")
\nFigure 7. All four aliases registered, each mapped to its openai/... target.
The master key can reach every model and create more keys. No application should\never hold it. Instead, issue a virtual key scoped to what that app actually needs.
\nVirtual Keys ā + Create New Key:
\n| Field | Value |
|---|---|
| Owned By | You |
| Team | leave empty |
| Key Name | demo-app |
| Models | qwen-small |
| Max Budget (USD) | 0.10 |
| Reset Budget | daily |
Two things about this form. Max Budget lives under Optional Settings, which\nstarts collapsed ā easy to miss and then wonder where the budget field went. And\nleaving Models empty means all models, the opposite of what you want here.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw-key-create-f440427f7d6f55ad01482ed469ff3624.png\")
\nFigure 8. Scoped to one model, capped at ten cents a day.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw-key-saved-e13c0abfcd4b4c0bfefa791764eaae50.png\")
\nFigure 9. The dialog says it plainly. Believe it.
The generated key is shown once and never again. Close this without copying and\nyour only option is to delete the key and create another.
Now the part that justifies the whole exercise. Same key, two models:
\ncurl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $VK\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"say hi\"}],\"max_tokens\":20}' | jq -r '.choices[0].message.content'
Hi there! How can I assist you today?
curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $VK\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-large\",\"messages\":[{\"role\":\"user\",\"content\":\"say hi\"}],\"max_tokens\":20}' | jq
{
\"error\": {
\"message\": \"key not allowed to access model. This key can only access models=['qwen-small']. Tried to access qwen-large\",
\"type\": \"key_model_access_denied\",
\"param\": \"model\",
\"code\": \"403\"
}
}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/gw-key-403-ff8ff7ee956b8b081dd67c4d3834ac10.png\")
\nFigure 10. One credential, one model. Leak it and the blast radius is that one line.
Compare that to the situation we started in: a key in five .env files, with access\nto all thirteen models and no way to tell which app is using it.
You can issue a scoped credential per application, restrict it to specific models\nwith a spend cap, and revoke it on its own without touching anything else.
Every call lands in a spend log:
\ncurl -s http://127.0.0.1:4000/spend/logs \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
| jq '[.[] | select(.spend > 0)] | .[-3:] | .[] | {model, spend, prompt_tokens, completion_tokens}'
{
\"model\": \"openai/Qwen3.5-9B\",
\"spend\": 8.520000000000001e-05,
\"prompt_tokens\": 16,
\"completion_tokens\": 20
}
{
\"model\": \"openai/Qwen3.5-9B\",
\"spend\": 2.62e-05,
\"prompt_tokens\": 11,
\"completion_tokens\": 5
}
![\"Spend](\"https://development-wec.wiline.com/docs/assets/images/gw-spend-logs-dfdc834679299ecd1fcfcc774e83b5b3.png\")
\nFigure 11. Tokens counted per call, with a price attached to each.
That select(.spend > 0) isn't cosmetic. Failed calls are logged too, at zero\ntokens and zero spend ā the 401s from earlier are all sitting in there with\nstatus: \"failure\". Useful on its own: the spend log doubles as a record of\nsomeone trying keys that don't work.
Those dollar figures are real. They match the published per-model pricing for the\nAPI this gateway is calling ā $0.70 per million input tokens and $3.70 per million\noutput for Qwen3.5-9B, the same rates you see listed against each model before\nyou pick one. So the spend log is a genuine record of what every call cost, broken\ndown per key and per model, from the first request.
Two more things in the full log entry, both useful:
\n\"litellm_overhead_time_ms\": 36.189,
\"messages\": {},
\"response\": {}
The gateway measures its own overhead and reports it per request ā 36ms here, which\nis a real number you can hold it to. And prompts and responses aren't stored by\ndefault: the log knows a call happened and what it cost, not what was said.
\nAlmost always the database. Check docker compose logs litellm for migration\nerrors ā if the gateway came up before Postgres was ready, the depends_on\ncondition is missing or the healthcheck isn't passing.
content comes back null with finish_reason: \"length\"āThe model is a reasoning model and it used your whole token budget thinking. Raising\nmax_tokens helps but doesn't guarantee anything ā the same prompt at 200 tokens\nanswered on one run and returned null on the next. For work that doesn't need\nreasoning, send it to a model that doesn't do any.
It's bound to 127.0.0.1, so it's unreachable from anywhere but the box itself.\nUse an SSH tunnel. If the tunnel is up and the page still won't load, confirm the\nport in docker compose ps matches the one you forwarded.
Chains joined with && stop at the first command that fails, and a failed grep\ncounts as a failure. Sourcing an env file whose variable name you guessed wrong\nleaves a placeholder in place and everything looks fine until an auth error three\nsteps later. Check for placeholders explicitly:
grep -c 'your-wec-key-here' .env
From the terminal, by alias:
\ncurl -s -X POST http://127.0.0.1:4000/key/delete \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"key_aliases\":[\"demo-app\"]}' | jq
You now have one endpoint, a credential per app, and a log of every call. The\nobvious question is the one Figure 4 raised: if the small model answers in 3 tokens\nwhat the reasoning model spends 200 on without answering at all, why is anything\nrouting to the expensive one by default?
\nThe next post puts the gateway in charge of that decision, and measures what asking\nit costs.
\nPart 1 of this series audited a live\nmulti-agent box and found every network-exposure question worth asking. One line from\nthat audit didn't get followed up: \"the Postgres, ClickHouse and Redis behind Langfuse\nwere published to 127.0.0.1 instead of the world ā someone made a good decision\nthere. Hold that thought.\"
Here's the other half of that thought: getting the network right says nothing about\nwhat happens after someone's already inside a container. If the process running in\nthere is root, a compromise starts with the keys to the whole filesystem. So we checked\nā on the same box, the same Langfuse stack Part 1 already praised ā whether \"network\ncorrect\" also meant \"privilege correct.\" It didn't, for two of the six containers. Here's\nwhat fixing that actually looked like, including the part that broke.
\n\nDocker doesn't tell you this by default ā a container can be published correctly,\nhealthy, and running as root the entire time, with nothing in docker ps hinting at\nit. The only way to know is to ask the container directly:
docker exec langfuse-postgres-1 id -u
docker exec langfuse-redis-1 id -u
0
0
![\"Both](\"https://development-wec.wiline.com/docs/assets/images/privsec-root-confirmed-9af63f21bf8c07c7554a6cbab4f02929.png\")
\nFigure 1. The check that most setups never run: both id -u calls come back 0.\nHealthy, published correctly, and root the entire time.
0 is root. Both were running as root, and looking at the compose file explains why ā\nneither service has a user: directive at all:
redis:
image: docker.io/redis:7
restart: always
command: >
--requirepass ${REDIS_AUTH:-myredissecret}
--maxmemory-policy noeviction
...
postgres:
image: docker.io/postgres:${POSTGRES_VERSION:-17}
restart: always
...
Nobody decided these should run as root. It's just what happens when you don't set\nanything ā same shape as Part 1's finding that an unset Redis password defaults to no\nauthentication at all. The clickhouse service in the same file shows the fix already\nhalf-done, sitting right there for comparison:
clickhouse:
image: docker.io/clickhouse/clickhouse-server
restart: always
user: \"101:101\"
One line. Somebody set it for ClickHouse and never got to the other two.
\nDon't invent a UID ā both official images already ship a non-root user built for exactly\nthis, and guessing wrong means either a permission error or, worse, a UID that silently\ndoesn't match the one the image's own files are owned by:
\ndocker exec langfuse-postgres-1 id postgres
docker exec langfuse-redis-1 id redis
uid=999(postgres) gid=999(postgres) groups=999(postgres),101(ssl-cert)
uid=999(redis) gid=999(redis) groups=999(redis)
Both images happen to use 999:999 for their built-in non-root user. Add it to the\ncompose file:
redis:
image: docker.io/redis:7
user: \"999:999\"
restart: always
...
postgres:
image: docker.io/postgres:${POSTGRES_VERSION:-17}
user: \"999:999\"
restart: always
...
docker compose up -d --force-recreate postgres redis
ā Container langfuse-postgres-1 Recreated
ā Container langfuse-redis-1 Recreated
sleep 5
docker compose ps
docker exec langfuse-postgres-1 id -u
docker exec langfuse-redis-1 id -u
langfuse-postgres-1 Up 34 seconds (healthy) 127.0.0.1:5433->5432/tcp
langfuse-redis-1 Up 34 seconds (healthy) 127.0.0.1:6379->6379/tcp
999
999
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/privsec-nonroot-healthy-2aa6142e51931837aa80ee308e8be448.png\")
\nFigure 2. Same two containers, seconds after --force-recreate: both healthy, both\nid -u now 999 instead of 0.
No permission errors, no ownership drama, both healthy within seconds. That's worth\nsaying plainly because it's not always true ā if these volumes had been initialized as\nroot somewhere upstream and never chown-ed, this exact command can fail with\npermission denied on the data directory. It didn't here, but check your own logs after\nrecreating, don't assume \"no output\" means \"no problem.\"
You can check whether a running container is actually root (docker exec <container> id -u), find its image's intended non-root user instead of guessing a UID\n(docker exec <container> id <username>), and convert it with a one-line user: in\ncompose.
Everything above was clean. Then the logs from a completely different container told a\ndifferent story:
\ndocker logs langfuse-langfuse-worker-1 --tail 20
Error: getaddrinfo ENOTFOUND redis
at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:122:26)
Redis error getaddrinfo ENOTFOUND redis
Redis error connect ECONNREFUSED 172.19.0.5:6379
Queue job mixpanel-integration-processing-queue errored: Error: Socket timeout.
Expecting data, but didn't receive any in 30000ms.
Queue job trace-delete errored: Error: Socket timeout. Expecting data, but didn't
receive any in 30000ms.
![\"langfuse-worker's](\"https://development-wec.wiline.com/docs/assets/images/privsec-worker-broken-e776f29e677755bd8a5e61864dbe22bc.png\")
\nFigure 3. A container nobody touched, failing anyway ā every queue job erroring with\nthe same 30-second socket timeout against Redis, well after the recreate that caused it.
langfuse-worker never touched Postgres or Redis's ownership ā it just holds a live\nconnection to Redis, and --force-recreate tore down the container that connection\npointed at. Docker's own healthcheck said Redis was healthy again well before the\nworker gave up retrying; ioredis just doesn't reconnect cleanly on its own here, and\nkept failing with 30-second socket timeouts long after the dependency it depended on was\nback.
The fix is the dependent service's own restart, not another look at Redis:
\ndocker compose restart langfuse-worker langfuse-web
2026-08-10T19:40:22.505Z info Redis connection has been closed.
2026-08-10T19:40:22.769Z info Shutdown complete, exiting process...
sleep 10
docker logs langfuse-langfuse-worker-1 --tail 15
experiment-create-queue executor started: true
posthog-integration-queue executor started: true
data-retention-queue executor started: true
webhook-queue executor started: true
Listening: http://21caf9a28775:3030
![\"langfuse-worker's](\"https://development-wec.wiline.com/docs/assets/images/privsec-worker-recovered-ca07f68b5e4cb0994524f20c497f41ca.png\")
\nFigure 4. Same container, one restart later ā every queue executor starts clean, no\nRedis errors anywhere in the log.
Clean queue startup, no Redis errors. That's the whole lesson: hardening a container\ndoesn't stay contained to that container. Anything holding a live connection to the\nthing you just recreated needs its own restart, on purpose, not as an afterthought you\ndiscover from an error log twenty minutes later.
\n--force-recreate on a shared dependency (a database, a queue, a cache) can silently\nbreak every service that already had a connection open to it ā even after the\ndependency itself reports healthy again. Restart dependents explicitly; don't assume\nthey'll notice on their own.
| Container | Before | Fix | Broke anything? |
|---|---|---|---|
langfuse-postgres-1 | root (uid 0) | user: \"999:999\" | No ā recreated clean |
langfuse-redis-1 | root (uid 0) | user: \"999:999\" | No ā recreated clean |
langfuse-clickhouse-1 | already 101:101 | none needed | ā |
langfuse-langfuse-worker-1 | untouched | none ā just needed a restart | Yes ā lost its Redis connection after the other two containers recreated |
Two one-line fixes. The actual cost wasn't the fix ā it was noticing the third\ncontainer that nobody touched was the one that broke.
\ndocker compose restart said \"no configuration file provided: not found\"āYou're not in the project directory Compose expects ā it only finds docker-compose.yml\nrelative to your current working directory (or via -f). cd into the stack's own\nfolder first; this exact error means Compose never even looked at your containers.
Restarting the dependency doesn't help ā the connection that broke belongs to the\ndependent service. Restart the thing holding the connection (here, langfuse-worker),\nnot the thing it connects to.
docker exec <container> id <username> returns \"no such user\"āThe image doesn't ship a dedicated non-root user under that name ā check the image's own\ndocumentation, or docker exec <container> cat /etc/passwd to see what's actually\navailable before picking a UID.
Privilege is the second layer: network hardening keeps a compromise from spreading\nbetween stacks, non-root keeps a compromise from owning the whole container. The last\nlayer in this series is identity ā putting real authentication in front of the ports\nthat must stay exposed, which is where NetBird\nand Caddy come in.
\nYou start with one agent. Then it needs a database. Then you add a second agent, a\nmessaging bridge, an observability stack. Six months later a single WEC Instance is running\nfive compose projects, twenty-something containers, and nobody remembers which ports are\nopen to the world.
\nThat's not a hypothetical ā that's the box this tutorial was written on. So instead of\ntheorizing, we probed it: can containers reach each other across stacks? Can they reach the\ndatabases? Is the firewall actually protecting anything?
\nThree of the answers surprised me. One of them was a database sitting there with no\npassword. And the firewall ā the firewall was lying.
\n\nFive independent compose projects, deployed over months, each with its own network (host, none and bridge are Docker's built-ins, plus one leftover from a service that isn't running):
docker network ls
NETWORK ID NAME DRIVER SCOPE
655850b24125 bridge bridge local
05e5d49e7e9d evolution-api_default bridge local
e8bd75ecbb0a host host local
ce697baac173 langfuse_default bridge local
ead3e0822ce6 nlp-evaluation-service_nlp-net bridge local
5d0ab32d14d7 none null local
9a6727150f2b openclaw_default bridge local
ec77b98f457b rag-service_default bridge local
5f88cabf97f0 remark42_default bridge local
And here's what's published to the world:
\ndocker ps --format '{{.Names}}\\t{{.Ports}}' | grep '0.0.0.0'
remark42-remark42-1 0.0.0.0:8082->8080/tcp, [::]:8082->8080/tcp
evolution-api-evolution-api-1 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
langfuse-langfuse-web-1 0.0.0.0:3001->3000/tcp, [::]:3001->3000/tcp
langfuse-minio-1 0.0.0.0:9090->9000/tcp, [::]:9090->9000/tcp, 127.0.0.1:9091->9001/tcp
![\"The](\"https://development-wec.wiline.com/docs/assets/images/netsec-exposure-audit-904ef03d2c7552d9b1b8c9d9e69218dd.png\")
\nFigure 1. The start of any audit. Nine networks: three Docker built-ins (bridge, host,\nnone), five compose projects, and one leftover. Four services published on 0.0.0.0 ā every\ninterface, including the public one.
Four services listening on every interface. Note what's not in that list: the Postgres,\nClickHouse and Redis behind Langfuse. Those were published to 127.0.0.1 instead ā someone\nmade a good decision there. Hold that thought.
The classic worry: OpenClaw gets compromised, and the attacker walks into Langfuse's\ndatabase next door. Let's find out instead of guessing. Grab the container IPs:
\nfor c in langfuse-postgres-1 evolution-api-evolution-postgres-1; do
docker inspect -f '{{.Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $c
done
/langfuse-postgres-1 172.19.0.5
/evolution-api-evolution-postgres-1 172.23.0.4
Now try to reach both from inside the OpenClaw container, by IP ā no DNS, no shortcuts:
\ndocker exec openclaw-openclaw-gateway-1 node -e '
const net=require(\"net\");
[[\"langfuse-postgres\",\"172.19.0.5\",5432],[\"evolution-postgres\",\"172.23.0.4\",5432]]
.forEach(([n,ip,p])=>{
const s=net.connect({host:ip,port:p,timeout:4000});
s.on(\"connect\",()=>{console.log(`REACHED ${n}`);s.destroy()});
s.on(\"timeout\",()=>{console.log(`blocked ${n}`);s.destroy()});
s.on(\"error\",e=>console.log(`error ${n} ${e.code}`));
});'
blocked langfuse-postgres
blocked evolution-postgres
![\"Cross-network](\"https://development-wec.wiline.com/docs/assets/images/netsec-isolation-blocked-52d0644c0dbd63427ca62b09eb5b9947.png\")
\nFigure 2. Connecting by raw IP from one stack's container to another stack's database:\nboth time out. Docker's isolation rules are doing their job.
Blocked. Modern Docker installs DOCKER-ISOLATION-STAGE rules that drop traffic between\nuser-defined bridge networks, and they work. If you've read that \"Docker networks don't\nreally isolate,\" test it on your own box before believing it ā on current Docker they do.
You can verify network isolation empirically instead of trusting folklore: get the target's\nIP with docker inspect, then connect by IP from inside another container.
So cross-stack isolation is fine. Which means the real risk is somewhere else.
\nEach compose project puts all its services on one network. That's the default, and it means\nthe application container can reach every backing service ā which is intended. The\nquestion is what that access actually looks like. From Evolution API's bridge container:
\ndocker exec evolution-api-bridge-1 python3 -c '
import socket
for name, host, port in [(\"postgres\",\"evolution-postgres\",5432),
(\"redis\",\"evolution-redis\",6379),
(\"api\",\"evolution-api\",8080)]:
s=socket.socket(); s.settimeout(4)
try:
s.connect((host,port)); print(f\"REACHED {name}:{port}\")
except Exception as e:
print(f\"blocked {name}:{port} -- {type(e).__name__}\")
finally: s.close()'
REACHED postgres:5432
REACHED redis:6379
REACHED api:8080
Reachable is expected. Unauthenticated is not. Check whether those services actually ask\nfor credentials ā first Redis:
\ndocker exec evolution-api-bridge-1 python3 -c '
import socket
s=socket.socket(); s.settimeout(5); s.connect((\"evolution-redis\",6379))
s.sendall(b\"INFO server\\r\\n\")
print(s.recv(200).decode(errors=\"replace\")[:120])'
$625
# Server
redis_version:7.4.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b61b4eb609520881
redis_
![\"Redis](\"https://development-wec.wiline.com/docs/assets/images/netsec-redis-noauth-22e25e2b5bbf007d514a914ec9ec8b91.png\")
\nFigure 3. The finding: INFO server answered with no credentials (output truncated at 120\nchars by the probe). Any container on this network can read or wipe the session store.
That's a INFO command answered with no AUTH ā the Redis holding this deployment's\nsession state has no password. Anything that gets code execution in any container on that\nnetwork can read it, write to it, or FLUSHALL it.
Postgres, in the same stack, behaves correctly:
\ndocker exec evolution-api-bridge-1 python3 -c '
import socket,struct
s=socket.socket(); s.settimeout(5); s.connect((\"evolution-postgres\",5432))
msg=b\"user\\x00postgres\\x00database\\x00postgres\\x00\\x00\"
s.sendall(struct.pack(\"!ii\",len(msg)+8,196608)+msg)
print(s.recv(64))'
b'R\\x00\\x00\\x00\\x17\\x00\\x00\\x00\\nSCRAM-SHA-256\\x00\\x00'
It demands SCRAM-SHA-256. Same box, same network, two backing services ā one asks for a\npassword, the other doesn't. Nobody decided that; it's just what the images default to when\nyou don't set a password.
\nThe lesson: \"internal network\" is not a security boundary. It's a convenience. Treat\nevery service as if the attacker is already on that network, because if one container falls,\nthey are.
\nNow the big one. Turn on UFW and deny everything inbound except SSH:
\nsudo ufw allow 22/tcp # ALWAYS first, or you lock yourself out
sudo ufw default deny incoming
sudo ufw --force enable
sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
60000:61000/udp ALLOW IN Anywhere
22/tcp ALLOW IN Anywhere
60000:61000/udp (v6) ALLOW IN Anywhere (v6)
22/tcp (v6) ALLOW IN Anywhere (v6)
Deny incoming ā only SSH and mosh's UDP range allowed. Now, from a different machine,\nopen the Langfuse UI and the Evolution API:
\nhttp://<instance-ip>:3001 -> Langfuse UI loads normally
http://<instance-ip>:8080 -> {\"status\":200,\"message\":\"Welcome to the Evolution API...\"}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/netsec-ufw-bypass-130569873d3680226c3534f016c4756a.png\")
Figure 4. The firewall says it denies everything inbound. The Evolution API on :8080\nanswers anyway ā status 200, straight from a browser on another machine.
Both wide open, through a firewall configured to block them. This is the single most common\nsecurity surprise in self-hosted Docker, and it isn't a bug.
\nLook at the order of the FORWARD chain:
sudo iptables -L FORWARD -n --line-numbers | head -8
Chain FORWARD (policy DROP)
num target prot opt source destination
1 DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
2 DOCKER-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
5 ufw-before-logging-forward all -- 0.0.0.0/0 0.0.0.0/0
6 ufw-before-forward all -- 0.0.0.0/0 0.0.0.0/0
![\"UFW](\"https://development-wec.wiline.com/docs/assets/images/netsec-ufw-chains-7a5553d8ce1882eb415c7e12e1beb065.png\")
\nFigure 5. The whole explanation in one screen: the firewall says deny incoming, and\nDocker's chains sit at positions 1-2 while UFW's start at 5.
Docker's chains sit at positions 1 and 2. UFW's don't appear until 5 and 6. And\npublishing a port isn't a listener on the host ā it's a NAT rule:
\nsudo iptables -t nat -L DOCKER -n | grep -E '3001|8080'
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 to:172.23.0.3:8080
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001 to:172.19.0.2:3000
The packet arrives, Docker's NAT rewrites the destination to the container, and it becomes\nforwarded traffic that Docker's own chain accepts ā long before any UFW rule is consulted.\nYour UFW rules aren't ignored; they're simply never reached.
\nufw deny does not protect a published Docker port. If you've been relying on UFW in\nfront of -p 8080:8080, that port has been open the whole time.
The cleanest fix isn't a firewall rule; it's not opening the door. If a service is only\nconsumed by other containers, it needs no ports: entry ā service-to-service traffic\nworks over the compose network by name.
If you need it reachable from the host (a debugging port, a local psql), bind it to\nloopback explicitly:
\nservices:
postgres:
# 127.0.0.1 prefix = host-only, never the public interface
ports:
- \"127.0.0.1:5433:5432\"
That's exactly what the Langfuse stack on this box already does ā its Postgres, ClickHouse\nand Redis are all on 127.0.0.1, which is why they never showed up in our exposure audit.\nThe default \"5433:5432\" means 0.0.0.0 ā every interface, including the public one.
internalāOne network per compose project is the default, not a design. Give the backing services\ntheir own network, and put only the application in both:
\nservices:
app: # talks to the world AND to the database
image: your-agent
networks: [frontend, backend]
db:
image: redis:7
command: redis-server --requirepass ${REDIS_PASSWORD}
networks: [backend] # backend only ā no route out
public: # e.g. a webhook receiver
image: your-public-thing
networks: [frontend] # cannot see the database at all
networks:
frontend:
backend:
internal: true # no gateway: no inbound, no outbound
Verified on the same box:
\n# the frontend-only container tries to reach the database
docker exec netsecdemo-public-1 python3 -c '
import socket
s=socket.socket(); s.settimeout(4)
try: s.connect((\"db\",6379)); print(\"REACHED db <- leak\")
except Exception as e: print(f\"blocked: {type(e).__name__}\")'
# the app, which is on both networks
docker exec netsecdemo-app-1 python3 -c '
import socket
s=socket.socket(); s.settimeout(4)
try: s.connect((\"db\",6379)); print(\"REACHED db <- correct, it needs it\")
except Exception as e: print(f\"blocked: {type(e).__name__}\")'
# and can the database itself reach the internet?
docker exec netsecdemo-db-1 timeout 6 getent hosts pypi.org >/dev/null \\
&& echo \"resolved external DNS\" || echo \"no internet egress <- internal working\"
blocked: gaierror
REACHED db <- correct, it needs it
no internet egress <- internal working
![\"The](\"https://development-wec.wiline.com/docs/assets/images/netsec-internal-demo-6231d6936686cf9d282c7f494797a185.png\")
Figure 6. Three properties proven in one run ā the public-facing container can't even\nresolve the database, the app can reach it, and the database has no route to the internet.
\nThree properties at once: the public-facing container can't even resolve the database, the\napp can, and the database can't phone home ā which matters the day a dependency of yours\nturns malicious.
\nOur audit found Redis answering INFO with no credentials. One line fixes it:
db:
image: redis:7
command: redis-server --requirepass ${REDIS_PASSWORD}
-NOAUTH Authentication required.
That's the real output from this box, after applying the fix ā the finding from Test 2 is\nclosed. If you set a password on a Redis that something already uses, remember to update the\nclient's connection string too (redis://:<password>@host:6379/0), or you'll trade an open\ndatabase for a broken one.
Do this even for services that aren't published. Defense in depth means the second layer\nholds when the first one fails.
\nDOCKER-USERāSometimes a port genuinely has to be public-facing but restricted by source. The chain to\nuse is DOCKER-USER ā Docker guarantees it runs before its own accept rules, and never\noverwrites it:
# only 203.0.113.0/24 may open new connections to the container's port 3000
sudo iptables -I DOCKER-USER 1 -p tcp --dport 3000 \\
-m conntrack --ctstate NEW ! -s 203.0.113.0/24 -j DROP
Tested live on the Langfuse port, from a browser on another machine:
\nhttp://<instance-ip>:3001 -> Langfuse UI loads
ERR_CONNECTION_TIMED_OUT
![\"The](\"https://development-wec.wiline.com/docs/assets/images/netsec-docker-user-blocked-8e34c54fd3b4552806e49bf87aed7cfb.png\")
Figure 7. Same URL, same firewall, one rule in the right chain ā and the port is finally\nclosed. Note Chrome's own suggestion: \"Checking the proxy and the firewall.\" This time the\nfirewall really is the answer.
\nMeanwhile the Evolution API on :8080, which we deliberately left alone, kept answering.\nThe rule is surgical, and it works exactly where UFW couldn't.
iptables -I is not persistent. Use iptables-persistent (netfilter-persistent save), or\nput the rule in a small systemd unit that runs after docker.service. A firewall rule that\ndisappears on reboot is worse than none, because you'll think you're protected.
You can audit and lock down a multi-stack Docker host: prove what's reachable, bind or split\nwhat shouldn't be, authenticate the backends, and filter published ports in the one chain\nDocker respects.
Run these four commands on any box you own. They take a minute and tell you more than any\narchitecture diagram:
\n# 1. What's exposed to the world?
docker ps --format '{{.Names}}\\t{{.Ports}}' | grep '0.0.0.0'
# 2. Are your firewall rules even in the path? (Docker chains before ufw = they aren't)
sudo iptables -L FORWARD -n --line-numbers | head -8
# 3. What can a compromised container reach on its own network?
docker exec <container> python3 -c \"import socket;s=socket.socket();s.settimeout(3);s.connect(('<svc>',<port>));print('reached')\"
# 4. Does that backing service ask for a password?
docker exec <container> python3 -c \"
import socket;s=socket.socket();s.settimeout(4);s.connect(('redis',6379))
s.sendall(b'INFO server\\r\\n');print(s.recv(60))\"
| Finding | Severity | Fix |
|---|---|---|
| Cross-network isolation works | ā none | nothing ā Docker already does this |
| Every service on one network, mutually reachable | ā ļø medium | split by trust zone, internal: true |
Redis answering with no AUTH | š“ high | one line: --requirepass |
ufw deny not applied to published ports | š“ high | bind to 127.0.0.1, or DOCKER-USER rule |
Backing DBs on 127.0.0.1 (Langfuse) | ā none | already correct ā copy this pattern |
None of the fixes took more than a line of YAML. The expensive part was knowing which line\nā and the only way to know is to probe your own box instead of trusting the diagram in your\nhead.
\nDOCKER-USER rule and nothing changedāTwo usual causes. First, --dport must be the container's port, not the published one ā\nthe DNAT already rewrote the destination by the time DOCKER-USER sees the packet (in our\ncase 3000, not 3001). Second, existing connections aren't affected: the\n--ctstate NEW match only touches new ones, so an already-open browser tab keeps working.
sudo ufw allow 22/tcp before ufw enable, always. If you're already locked out, most\nproviders give you a serial/VNC console ā use it to run ufw disable.
internal: true broke my container's package installāThat's it working. An internal network has no gateway, so no egress at all ā no pip install, no DNS, no calling an external API. Backing services shouldn't need any of that;\nif a container does, it belongs on the frontend network too.
Compose DNS only resolves names within a shared network. If app and db are on\ndifferent networks with nothing in common, db won't resolve. The app has to be a member of\nboth, as in Fix 2.
Network hardening is the layer that keeps a compromised container from becoming a\ncompromised host. The next layers, in the order I'd do them: run containers as non-root\n(user: in compose) so a breakout starts with fewer privileges, and put an identity\nprovider in front of the ports you must expose ā a private mesh with\nNetBird plus authentication is the natural pairing\nwith everything above.
![\"Hermes](\"https://development-wec.wiline.com/docs/assets/images/hermesagent-c76bbe81610ae2e70b0950e0f96bec8a.png\")
+In Part 1 you self-hosted Hermes with\npersistent memory. This part connects it to the same Telegram bot from the\nOpenClaw series ā the chat your\nusers already know keeps working exactly as it did, just a different agent\nanswering underneath. No new bot to announce, no channel to migrate people to.
\n\nContinues on the same WEC Instance as Part 1. Hermes gateway is already\ninstalled and running as a systemd user service ā this part is configuration,\nnot installation. Every command and error below is from the actual run.
Self-hosting is only worth the effort if it doesn't lock you into whatever\nagent you started with. The bot itself ā its name, its chat history, the\npeople who already talk to it ā is the asset. The agent behind it is\nreplaceable. Swapping OpenClaw for Hermes on the same bot proves that: same\nTelegram front door, same conversation thread, new brain.
\nhermes gateway setup
Pick Telegram from the platform list, then choose how to connect it:
\nā Telegram
How would you like to create your Telegram bot?
[1] Automatic (recommended)
Scan a QR code ā confirm in Telegram ā done.
No token copy-paste needed.
[2] Manual
Create a bot via @BotFather yourself and paste the token.
Choice [1/2] [1]: 2
Create a bot via @BotFather on Telegram
Telegram bot token: **********************************************
ā Telegram token saved
Take option 2 ā option 1 spins up a brand new bot via QR code, which\ndefeats the point if you're reusing an existing one.
\nNext the wizard asks who's allowed to use the bot:
\nš Security: Restrict who can use your bot
To find your Telegram user ID:
1. Message @userinfobot on Telegram
2. It will reply with your numeric ID (e.g., 123456789)
Allowed user IDs (comma-separated, leave empty for open access):
If you don't have your ID handy, leave it blank and move on ā you can lock\nthis down later. But the prompt is misleading: leaving it empty doesn't\nopen access by default. The gateway logs told the real story after restart:
\nWARNING gateway.run: No user allowlists configured. All unauthorized users
will be denied. Set GATEWAY_ALLOW_ALL_USERS=true in ~/.hermes/.env to allow
open access, or configure platform allowlists (e.g.,
TELEGRAM_ALLOWED_USERS=your_id).
An empty allowlist means deny everyone, not open access. If you want open\naccess (fine for a personal bot, not for anything public-facing), you need\none more step:
\necho 'GATEWAY_ALLOW_ALL_USERS=true' >> ~/.hermes/.env
The wizard also asks for a Home Channel ā where Hermes delivers cron\nresults and cross-platform messages. Leave it blank too; you can set it later\nfrom inside the chat with /sethome.
Finish the wizard and let it restart the gateway:
\nRestart the gateway to pick up changes? [Y/n]: Y
ā User service restarted
![\"Terminal](\"https://development-wec.wiline.com/docs/assets/images/hermes-gateway-setup-wizard-7293ce9021ee147b93bad2dcb7a72a53.png\")
\nFigure 1. The whole wizard, start to finish ā token, the allowlist warning, and the restart.
If you added GATEWAY_ALLOW_ALL_USERS after the wizard already restarted\nonce (as above), restart again so it picks up the .env change:
systemctl --user restart hermes-gateway.service
Message your bot. Not @BotFather ā your bot, by the username you gave it\nwhen you created it. It's an easy mix-up: BotFather answers everything with\nits own fixed command menu, so if you see that instead of a real reply,\nyou're in the wrong chat.
\nThe first message may get a system nudge instead of a chat reply ā that's\nHermes noticing there's no home channel yet, not a failure:
\nš¬ No home channel is set for Telegram. A home channel is where Hermes
delivers cron job results and cross-platform messages.
Type /sethome to make this chat your home channel, or ignore to skip.
Send another message and it answers for real:
\n![\"Telegram](\"https://development-wec.wiline.com/docs/assets/images/hermes-telegram-verified-17c22677d5e1e6a1d6147099b58fdc53.png\")
\nFigure 2. The same bot from the OpenClaw series, now answering through Hermes.
Ask it something with actual depth, and it delivers a real, structured answer\nā not just a canned reply:
\n![\"Telegram](\"https://development-wec.wiline.com/docs/assets/images/hermes-telegram-sky-blue-139494c527c4075440bf7afa25c14039.png\")
\nFigure 3. Same agent, a real question ā full explanation, not a one-liner.
Same bot, same chat history, new agent.
\n\"No user allowlists configured. All unauthorized users will be denied.\" ā\nleaving the allowlist prompt empty does not mean open access; it means\ndeny-all. Set GATEWAY_ALLOW_ALL_USERS=true in ~/.hermes/.env and restart\nthe gateway.
\"Telegram polling conflict... make sure that only one bot instance is\nrunning\" ā shows up for a few seconds right after a restart, while\nTelegram's servers finish releasing the previous session. It retries and\nclears on its own within 20 seconds; no action needed unless it keeps\nrepeating past the 5th retry, which means another process really is polling\nthe same token.
\nBot doesn't respond at all ā check you're messaging the bot itself, not\n@BotFather. BotFather always answers with its own command list regardless of\nwhat you send it.
\nThe bot answers ā next is putting it on a schedule: a morning briefing or a\nservice-health check delivered to this same chat, no polling required.
\n![\"SearXNG\"](\"https://development-wec.wiline.com/docs/assets/images/searxng-mark-1b536bb949960590cea598d56becda08.png\")
+Say you've built a support chatbot or a RAG assistant on WEC. It's solid on your docs and\non what the model already knows ā but ask it something current (\"what's the latest\nrelease of X?\", today's pricing, a recent change) and it either guesses or tells you its\nknowledge is out of date. For anything that needs to stay current, that's a real gap.
\nWEC Inference now has a web-search tool built in. Add it to a normal\n/v1/chat/completions call and the model can look things up on the live web ā the search\nruns on WiLine's own infrastructure (SearXNG), so nothing goes out to a third-party search\nvendor. Let's test it.
Calls hit https://inference.wiline.com/v1/chat/completions with your WEC Inference API\nkey, model Qwen3.5-9B. Nothing to install ā the tool is part of the API.
A normal call, asking something recent:
\ncurl -sS https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{
\"model\": \"Qwen3.5-9B\",
\"messages\": [{\"role\": \"user\", \"content\": \"What is the latest stable version of Docker Engine, and when was it released?\"}]
}'
![\"The](\"https://development-wec.wiline.com/docs/assets/images/websearch-without-aeaee1a5d9a326ccb92d56a8ec08bdc5.png\")
\nFigure 1. With no tool, the model won't commit to a version ā it says its knowledge is\nlimited to its training cutoff and points you to the release notes instead.
The same request, plus a web_search tool and tool_choice: auto:
curl -sS https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{
\"model\": \"Qwen3.5-9B\",
\"messages\": [{\"role\": \"user\", \"content\": \"What is the latest stable version of Docker Engine, and when was it released?\"}],
\"tools\": [{
\"type\": \"function\",
\"function\": {
\"name\": \"litellm_web_search\",
\"description\": \"Search the web for current information\",
\"parameters\": {
\"type\": \"object\",
\"properties\": {\"query\": {\"type\": \"string\", \"description\": \"The search query\"}},
\"required\": [\"query\"]
}
}
}],
\"tool_choice\": \"auto\"
}'
![\"The](\"https://development-wec.wiline.com/docs/assets/images/websearch-with-ae28b4b9dbd33effc4676a36ce37a4e0.png\")
\nFigure 2. Same model, same question, plus the tool ā now it returns a concrete version\nand release date pulled from the live web. The model decided to search, WEC ran the query,\nand fed the results back before answering.
That's the whole feature: one tool in the request body, current answers out.
\nWeb search injects the results into your prompt, so a searched call costs more tokens than\na bare one (in our test, prompt tokens went from ~26 to ~2,776). Keep tool_choice: auto\nso the model only searches when it actually needs to ā it'll skip search for questions it\ncan already answer.
auto handles that for you.Point it at a RAG assistant or a chatbot and it can stay current without you wiring up a\nseparate search service.
\nAn agent that calls tools does two very different things: it reasons (\"I should\nsearch the docs\") and it acts (actually calls the tool). When something goes wrong,\nthe first question is always which layer failed ā did it think wrong, or did the doing\nbreak? A flat log can't answer that. A trace can.
\nIn this tutorial you build a small tool-calling agent on WEC Inference, instrument it\nwith Langfuse so every reasoning step and every tool call becomes an inspectable node,\nthen debug two real failures from the trace tree ā including the worst kind: a\nconfident, wrong answer that never throws an error. Every command, error, and screenshot\nbelow is from a real run.
\n\nBuilt on a WEC Instance with Docker, against WEC Inference (Qwen3.5-122B) and a\nself-hosted Langfuse from the\nobservability tutorial. The agent's tools\ninclude the RAG service from the\nRAG tutorial ā so this piece ties the\nwhole series together. All model calls stay on WEC; nothing leaves for a third-party API.
The dotted lines are the point: every step reports itself to Langfuse, so the agent's\nreasoning and its actions land as separate, inspectable nodes.
\nPrerequisites: a WEC Instance with Docker, a WEC Inference API key, a running\nLangfuse (public + secret key, host URL), and a tool the agent can call ā here the RAG\n/ask service. All commands run in ~/agent-tracing.
Rule: never instrument before you've seen the raw behavior. So v1 does the minimum ā\nask the model a question with a tool definition, and print what comes back. Set up the\nworkspace and the WEC key:
\nmkdir -p ~/agent-tracing && cd ~/agent-tracing
grep '^WEC_API_KEY=' ~/evolution-api/.env > .env
agent.py (v1):
import os, requests
WEC_URL = \"https://inference.wiline.com/v1/chat/completions\"
WEC_API_KEY = os.environ[\"WEC_API_KEY\"]
MODEL = \"Qwen3.5-122B\"
TOOLS = [{
\"type\": \"function\",
\"function\": {
\"name\": \"search_docs\",
\"description\": \"Search the WEC documentation for an answer to a question.\",
\"parameters\": {
\"type\": \"object\",
\"properties\": {\"query\": {\"type\": \"string\"}},
\"required\": [\"query\"],
},
},
}]
def call_model(messages):
r = requests.post(WEC_URL,
headers={\"Authorization\": f\"Bearer {WEC_API_KEY}\"},
json={\"model\": MODEL, \"messages\": messages, \"tools\": TOOLS, \"tool_choice\": \"auto\"},
timeout=120)
r.raise_for_status()
return r.json()[\"choices\"][0][\"message\"]
if __name__ == \"__main__\":
msg = call_model([{\"role\": \"user\", \"content\": \"How do I create a compute instance on WEC?\"}])
print(\"=== REASONING LAYER (what it thought) ===\")
print(msg.get(\"reasoning_content\") or \"(none)\")
print(\"\\n=== ACTION LAYER (what it decided to do) ===\")
for tc in msg.get(\"tool_calls\") or []:
print(f\"tool: {tc['function']['name']} args: {tc['function']['arguments']}\")
Run it in a container (same pattern as the rest of the series):
\nFROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir requests
COPY agent.py .
CMD [\"python\", \"agent.py\"]
services:
agent:
build: .
env_file: .env
docker compose run --rm --build agent
![\"The](\"https://development-wec.wiline.com/docs/assets/images/agent-reasoning-action-56b6f9e7c5b8aef031cd06c04187e291.png\")
\nFigure 1. WEC Inference returns the two layers in one response: reasoning_content\n(the thinking) and tool_calls (the decision). We don't have to infer the split ā the API\nhands it to us.
Qwen3.5-122B returns finish_reason: tool_calls with a proper tool_calls array, plus\na reasoning_content field containing the model's chain of thought before it acts. That\nreasoning field is exactly what makes the reasoning-vs-action split visible later.
v1 decided to search but didn't. Now actually call the tool (the RAG /ask service),\nfeed the result back, and let the model write the final answer. Add the tool + the loop:
import json
RAG_URL = \"http://<your-vm-ip>:8000/ask\" # the RAG service = our one tool
def search_docs(query):
r = requests.post(RAG_URL, json={\"question\": query}, timeout=120)
r.raise_for_status()
return r.json()[\"answer\"]
def run(user_input):
messages = [{\"role\": \"user\", \"content\": user_input}]
while True:
msg = call_model(messages)
if not msg.get(\"tool_calls\"):
print(\"\\n=== FINAL ANSWER ===\\n\" + (msg.get(\"content\") or \"\"))
return msg.get(\"content\")
messages.append({\"role\": \"assistant\", \"content\": msg.get(\"content\") or \"\",
\"tool_calls\": msg[\"tool_calls\"]})
for tc in msg[\"tool_calls\"]:
args = json.loads(tc[\"function\"][\"arguments\"])
print(f\"[reasoning] {(msg.get('reasoning_content') or '').strip()[:140]}\")
print(f\"[action] {tc['function']['name']}({args})\")
result = search_docs(args[\"query\"])
print(f\"[tool result] {result[:140]}...\")
messages.append({\"role\": \"tool\", \"tool_call_id\": tc[\"id\"], \"content\": result})
docker compose run --rm --build agent
![\"The](\"https://development-wec.wiline.com/docs/assets/images/agent-loop-b09953660869437421c9222da3f9b135.png\")
\nFigure 2. The complete loop ā the model reasons, calls search_docs, gets the RAG\nresult, and writes a grounded answer. It works. But a working terminal tells you nothing\nabout whether the reasoning was sound ā that's what we fix next.
The terminal flattens everything into one stream. Langfuse turns each function into a\nnode so you can see the structure. Reuse the Langfuse keys from the observability\ntutorial (same project as your RAG traces):
\ngrep -E '^LANGFUSE_(PUBLIC_KEY|SECRET_KEY|HOST)=' ~/rag-service/.env >> .env
Add the SDK (pip install ... langfuse) and decorate three functions ā that's the whole\ninstrumentation. @observe wraps a function as a span; as_type=\"generation\" marks the\nLLM calls so Langfuse captures model, tokens, and cost:
from langfuse import observe, get_client
@observe(as_type=\"generation\")
def call_model(messages):
...
data = r.json()
get_client().update_current_generation(model=MODEL, usage_details=data.get(\"usage\"))
return data[\"choices\"][0][\"message\"]
@observe()
def search_docs(query):
...
@observe()
def run(user_input):
...
if __name__ == \"__main__\":
run(\"How do I create a compute instance on WEC?\")
get_client().flush() # short-lived script: push traces before exit
docker compose run --rm --build agent
Open Langfuse ā Tracing ā Traces. First thing you notice: there are two traces per\nquestion.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/trace-list-3a0d985919db7d12160c4109ab85dede.png\")
\nFigure 3. Distributed tracing, for free: your agent's run trace and the RAG service's\nown ask trace are separate ā each service instruments itself. Seeing both is how you\nfollow one request across service boundaries.
Open the newest run:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/trace-tree-green-c19c35d6b1ba42189355bdd0e79d96f8.png\")
\nFigure 4. The agent turn as a tree: run ā call_model (decide) ā search_docs\n(act) ā call_model (answer). Langfuse also captured latency, token usage\n(801ā306), and a correctness score ā all for three decorators.
Click the first call_model and open its output:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/trace-reasoning-content-c5e9876f6e9a640d45e9f5f54c5594cc.png\")
\nFigure 5. The reasoning layer, captured: reasoning_content (\"I should search the WEC\ndocumentationā¦\") sits right next to the tool_calls it produced. This is the record you\ndebug against.
Plant a realistic bug: the RAG API expects {\"question\": ...}, but it's natural to\nsend {\"query\": ...} because the tool's parameter is named query. One wrong key:
sed -i 's/{\"question\": query}/{\"query\": query}/' agent.py
docker compose run --rm --build agent
It crashes:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/agent-422-crash-af97e2fe3ae5709ec3fe6d5f859095a0.png\")
\nFigure 6. The terminal gives you a stack trace ā it tells you where the code\nbroke. It does not tell you whether the agent thought correctly. For that, go to the\ntrace.
Open the failed run and click the red search_docs span:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/trace-error-span-ddca621bf15a8cc64c96371cd6a42e1a.png\")
\nFigure 7. The whole diagnosis in one picture: call_model is green (the reasoning\nwas right ā it correctly chose to search), search_docs is red, died in 0.04s, and\nthe panel shows the 422 plus the exact payload it sent. An instant failure is a\nrejection, not a timeout ā pointing straight at a bad request.
Fix the one line and confirm green:
\nsed -i 's/{\"query\": query}/{\"question\": query}/' agent.py
docker compose run --rm --build agent
![\"The](\"https://development-wec.wiline.com/docs/assets/images/trace-fixed-green-c3b4677676dc40f861fca6dafe1594bb.png\")
\nFigure 8. Back to green ā search_docs succeeds and the final answer is produced. You\ndiagnosed and verified the fix from the trace.
That was the loud failure ā it crashed, so you'd have caught it eventually even without\na trace. The dangerous one is next.
\nReal agents have more than one tool, and the interesting decisions happen when the model\npicks between them. Add a get_pricing tool alongside search_docs, and route calls\nthrough a dispatch table keyed by tool name:
TOOLS = [
{\"type\": \"function\", \"function\": {
\"name\": \"search_docs\",
\"description\": \"Search the WEC documentation for how-to and setup questions.\",
\"parameters\": {\"type\": \"object\", \"properties\": {\"query\": {\"type\": \"string\"}}, \"required\": [\"query\"]}}},
{\"type\": \"function\", \"function\": {
\"name\": \"get_pricing\",
\"description\": \"Get the price of a WEC resource (e.g. 'compute instance', 'block storage', 'inference').\",
\"parameters\": {\"type\": \"object\", \"properties\": {\"resource\": {\"type\": \"string\"}}, \"required\": [\"resource\"]}}},
]
PRICES = {
\"compute instance\": \"$0.03 / vCPU-hour\",
\"block storage\": \"$0.10 / GB-month\",
\"inference\": \"$0.50 per 1M tokens\",
}
@observe()
def get_pricing(resource):
return PRICES.get(resource.lower().strip(), f\"No pricing found for '{resource}'.\")
TOOL_FUNCS = {\"search_docs\": search_docs, \"get_pricing\": get_pricing}
# in run(), dispatch by the tool the model actually chose:
for tc in msg[\"tool_calls\"]:
name = tc[\"function\"][\"name\"]
args = json.loads(tc[\"function\"][\"arguments\"])
print(f\"[action] {name}({args})\")
result = str(TOOL_FUNCS[name](**args))
messages.append({\"role\": \"tool\", \"tool_call_id\": tc[\"id\"], \"content\": result})
Ask a pricing question (run(\"How much does a compute instance cost on WEC?\")) and run\nit. The agent should pick get_pricing, not search_docs:
![\"Langfuse](\"https://development-wec.wiline.com/docs/assets/images/trace-two-tools-ba891f9c0ec4d99aea2e499dcbba3189.png\")
\nFigure 9. Genuine tool selection: given two tools, the agent chose get_pricing and\nanswered $0.03/vCPU-hour, Correctness 1.00. The choice itself is now a thing you can\nsee in the trace.
Now a bug that doesn't crash. Someone refactors the dispatch and hardcodes the old\nsingle tool, forgetting the new one ā so every call runs search_docs no matter what the\nmodel chose:
sed -i 's/result = str(TOOL_FUNCS\\[name\\](\\*\\*args))/result = str(search_docs(list(args.values())[0])) # BUG/' agent.py
docker compose run --rm --build agent
[action] get_pricing({'resource': 'compute instance'})
=== FINAL ANSWER ===
Based on the pricing information available, the highest cost compute instance category on
WEC is $1,586.77. ...
Read that carefully. The model chose get_pricing (correct). The answer is confident,\ndetailed ā and completely wrong (the real price is $0.03/vCPU-hour). There is no\nerror, no traceback. A log would look perfectly healthy. This is the failure mode that\nships to production and quietly lies to users.
The trace is the only thing that catches it. Open the run:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/trace-silent-failure-162a3ce9394bad7caca0e908be683b67.png\")
\nFigure 10. Three tells a log can't show: (1) the executed span is search_docs ā\nthere's no get_pricing span, even though the model chose it: decision ā action;\n(2) the reasoning node scored Correctness 0.00 ā the LLM-judge auto-flagged the wrong\nanswer; (3) the output shows the model confused by a tool result that didn't match what\nit asked for. Fix the dispatch back to TOOL_FUNCS[name](**args) and the price is correct\nagain.
You instrumented a tool-calling agent so its reasoning and its actions are separate,\ninspectable trace nodes ā and debugged two real failures from the trace: the loud\ncrash and the silent, confident-but-wrong answer that no stack trace would ever reveal.
get_client().flush() before the process ends (already in v3).search_docs span missing. Only @observe-decorated functions become nodes ā\ndecorate the function, not the call site.usage_details=data.get(\"usage\") via\nupdate_current_generation; without it Langfuse can't compute cost.tool_calls (what it chose)\nagainst the executed spans (what actually ran). A mismatch is a dispatch/routing bug ā and\na correctness score on the trace will flag it automatically.You can now see an agent think, act, and fail quietly ā and catch it. Point the same\ninstrumentation at a real multi-step agent, add automated correctness scoring on every\ntrace, and you have production observability that surfaces silent failures before your users\ndo.
\ncd ~/agent-tracing && docker compose down --rmi local
![\"WhatsApp\"](\"https://development-wec.wiline.com/docs/assets/images/whatsapp_logo-cc34546fcff90d36128c7230c48c496f.png\"){kind=logo}
++![\"Evolution\"](\"https://development-wec.wiline.com/docs/assets/images/0_hT2ECga7Z67dEbYz-62f9ab60d339ae8868fee46475e505bc.png\")
Most \"self-host a WhatsApp AI\" guides stop at \"the container started.\" This one goes all\nthe way: you deploy a real, programmable WhatsApp gateway (Evolution API), then\nwrite the bridge yourself ā the ~50 lines that turn an incoming message into an LLM\nanswer and send it back. That bridge (webhook ā model ā reply) is the reusable pattern\nbehind every chat-AI integration: SMS, Slack, Telegram, voice ā swap the channel, the\nshape is identical.
\nAnd because this is a real build, we hit ā and fix ā every gotcha: an image that moved\npublishers, a Baileys version loop, an infinite reply loop, group-chat spam, WhatsApp's\nnew LID addressing, and a genuine delivery wall that most tutorials pretend\ndoesn't exist. Every command, error, and output below is from an actual run.
\n\nEvolution links to a WhatsApp number as a companion device (like WhatsApp Web) ā the\nbot acts as that account and can read every DM and group it receives. For a demo on a\nnumber you control it's fine; for anything real, use a dedicated number. And see the\ndelivery limitation at the end before you build on this in\nproduction.
Four containers: Evolution + its Postgres + Redis, and the bridge you'll write. The\nbridge is the whole point ā everything else is off-the-shelf.
\nPrerequisites: a WEC Instance with Docker + Compose, a WEC Inference API key\n(Inference ā API Keys), a WhatsApp number for\nthe demo, and ~2 GB free disk.
\nEvolution needs its own Postgres and Redis. We give it a dedicated set on an internal\nnetwork and expose only the API port (8080). docker-compose.yml:
services:
evolution-api:
image: evoapicloud/evolution-api:v2.2.3
restart: unless-stopped
ports:
- \"8080:8080\"
env_file: .env
volumes:
- evolution_instances:/evolution/instances
depends_on: [evolution-postgres, evolution-redis]
logging:
driver: json-file
options: { max-size: \"50m\", max-file: \"3\" }
evolution-postgres:
image: postgres:17
restart: unless-stopped
environment:
POSTGRES_USER: evolution
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: evolution
volumes:
- evolution_pgdata:/var/lib/postgresql/data
evolution-redis:
image: redis:7
restart: unless-stopped
volumes:
- evolution_redis:/data
volumes:
evolution_instances:
evolution_pgdata:
evolution_redis:
The logging: block caps container logs at 3Ć50 MB. Skip it and a chatty container can\nfill the disk and take the box down ā a lesson learned the hard way on this VM. Set it\nbefore you need it.
The .env (generated secrets, connection URIs, and ā importantly ā a pinned Baileys\nWhatsApp-Web version):
cd ~/evolution-api && cat > .env << EOF
AUTHENTICATION_API_KEY=$(openssl rand -hex 24)
SERVER_URL=http://<your-vm-ip>:8080
POSTGRES_PASSWORD=$(openssl rand -hex 16)
DATABASE_ENABLED=true
DATABASE_PROVIDER=postgresql
DATABASE_CONNECTION_URI=postgresql://evolution:\\${POSTGRES_PASSWORD}@evolution-postgres:5432/evolution?schema=public
CACHE_REDIS_ENABLED=true
CACHE_REDIS_URI=redis://evolution-redis:6379/0
CACHE_REDIS_PREFIX_KEY=evolution
CACHE_LOCAL_ENABLED=false
QRCODE_LIMIT=30
EOF
Bring it up:
\ndocker compose up -d
sleep 15
curl -s http://localhost:8080 | jq
{
\"status\": 200,
\"message\": \"Welcome to the Evolution API, it is working!\",
\"version\": \"2.2.3\",
...
}
![\"Evolution](\"https://development-wec.wiline.com/docs/assets/images/wa-evolution-up-680a7f5fe9e4a450cacf33f1df5ed6f7.png\")
\nFigure 1. The stack is up: Evolution answers on :8080 with its version and status.
(1) The publisher moved. The image most guides reference ā atendai/evolution-api ā\nnow returns \"pull access denied / repository does not exist.\" The current image is\nevoapicloud/evolution-api. The old repo's tag list still resolves, which sends you\ndown a rabbit hole; confirm with a fresh pull of hello-world that it's not rate-limiting.\n(2) Pin a real tag. v2.1.1 (from a popular blog) was never published ā check\nhttps://hub.docker.com/v2/repositories/evoapicloud/evolution-api/tags and pin one that\nexists (we use v2.2.3).
Set your API key as a shell var (every request needs it in an apikey: header):
APIKEY=$(grep '^AUTHENTICATION_API_KEY=' .env | cut -d= -f2)
Create the instance:
\ncurl -s -X POST http://localhost:8080/instance/create \\
-H \"apikey: $APIKEY\" -H \"Content-Type: application/json\" \\
-d '{\"instanceName\":\"wec-demo\",\"integration\":\"WHATSAPP-BAILEYS\",\"qrcode\":true}' | jq '.instance'
Then open the built-in Manager at http://<your-vm-ip>:8080/manager, enter your server\nURL + API key, click wec-demo, and scan the QR from WhatsApp ā Settings ā Linked\nDevices ā Link a Device. Confirm it's connected:
curl -s http://localhost:8080/instance/fetchInstances -H \"apikey: $APIKEY\" \\
| jq '.[] | {name, connectionStatus}'
{ \"name\": \"wec-demo\", \"connectionStatus\": \"open\" }
![\"Evolution](\"https://development-wec.wiline.com/docs/assets/images/wa-manager-connected-d91fda82b98c6238d0d747b47636645b.png\")
\nFigure 2. The Evolution Manager dashboard once the instance links: Connected, with live contact, chat, and message counts.
If the instance never leaves connecting and the logs show ChannelStartupService\nre-initializing every few seconds with a Baileys version env: 2,3000,... line, the\npinned WhatsApp-Web version is stale and WhatsApp rejects the handshake before a QR is\never generated. Fetch the current version and pin it, then recreate:
curl -s \"https://raw.githubusercontent.com/WhiskeySockets/Baileys/master/src/Defaults/baileys-version.json\"
# {\"version\":[2,3000,1035194821]}
echo \"CONFIG_SESSION_PHONE_VERSION=2.3000.1035194821\" >> .env
docker compose up -d --force-recreate evolution-api
Now the part you actually write. Rule: never wire logic before you've seen the real\ndata. So v1 does nothing but print whatever Evolution sends.
\nThe bridge runs as another container in the same Compose, so Evolution reaches it by\nname (http://bridge:8090) and the bridge reaches Evolution at\nhttp://evolution-api:8080. Create bridge/main.py:
from fastapi import FastAPI, Request
app = FastAPI()
@app.post(\"/webhook\")
async def webhook(request: Request):
data = await request.json()
print(\"=== WEBHOOK RECEIVED ===\", flush=True)
print(data, flush=True)
return {\"received\": True}
flush=True mattersWithout it, print output is buffered and never shows in docker compose logs ā you'll\nstare at an empty log convinced it's broken.
bridge/Dockerfile:
FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi uvicorn requests
COPY main.py .
CMD [\"uvicorn\", \"main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8090\"]
Add the service to docker-compose.yml (inside services:), then build it:
bridge:
build: ./bridge
restart: unless-stopped
env_file: .env
depends_on: [evolution-api]
logging:
driver: json-file
options: { max-size: \"10m\", max-file: \"3\" }
docker compose up -d --build bridge
Now register the webhook so Evolution POSTs messages to the bridge:
\ncurl -s -X POST \"http://localhost:8080/webhook/set/wec-demo\" \\
-H \"apikey: $APIKEY\" -H \"Content-Type: application/json\" \\
-d '{\"webhook\":{\"enabled\":true,\"url\":\"http://bridge:8090/webhook\",\"webhookByEvents\":false,\"events\":[\"MESSAGES_UPSERT\"]}}' | jq '.enabled, .url'
Tail the logs (docker compose logs -f bridge) and send yourself a WhatsApp message. The\nreal payload appears:
{'event': 'messages.upsert', 'instance': 'wec-demo',
'data': {'key': {'remoteJid': '123456789012345@lid', 'fromMe': False, 'id': '...'},
'pushName': 'Test Contact',
'message': {'conversation': 'Oii'},
'messageType': 'conversation'},
'sender': '5511988887777@s.whatsapp.net', ...}
Three things we now know ā and every one drives the code next:
\ndata.message.conversation (plain) or data.message.extendedTextMessage.text (quoted)data.key.fromMe ā True for our own messages (the loop guard)remoteJid ends in @lid ā WhatsApp's privacy identifier, not a phone number (this one's a saga; Step 5)Give the bridge the WEC Inference key so it can call the model:
\necho \"WEC_API_KEY=<your-wec-inference-key>\" >> ~/evolution-api/.env
Now the logic ā main.py v2:
import os, requests
from fastapi import FastAPI, Request
app = FastAPI()
EVOLUTION_URL = \"http://evolution-api:8080\"
INSTANCE = \"wec-demo\"
EVOLUTION_APIKEY = os.environ[\"AUTHENTICATION_API_KEY\"]
WEC_API_KEY = os.environ[\"WEC_API_KEY\"]
WEC_URL = \"https://inference.wiline.com/v1/chat/completions\"
MODEL = \"Qwen2.5-3B-Instruct\"
def ask_llm(text: str) -> str:
r = requests.post(WEC_URL,
headers={\"Authorization\": f\"Bearer {WEC_API_KEY}\"},
json={\"model\": MODEL, \"messages\": [{\"role\": \"user\", \"content\": text}]},
timeout=60)
r.raise_for_status()
return r.json()[\"choices\"][0][\"message\"][\"content\"]
def send_whatsapp(number: str, text: str):
r = requests.post(f\"{EVOLUTION_URL}/message/sendText/{INSTANCE}\",
headers={\"apikey\": EVOLUTION_APIKEY, \"Content-Type\": \"application/json\"},
json={\"number\": number, \"text\": text}, timeout=60)
print(f\"SEND -> {r.status_code}\", flush=True)
@app.post(\"/webhook\")
async def webhook(request: Request):
data = (await request.json()).get(\"data\", {})
key = data.get(\"key\", {})
if key.get(\"fromMe\"): # (1) LOOP GUARD
return {\"skipped\": \"fromMe\"}
msg = data.get(\"message\", {}) # (2) extract text
text = msg.get(\"conversation\") or msg.get(\"extendedTextMessage\", {}).get(\"text\")
if not text:
return {\"skipped\": \"no text\"}
to = key.get(\"remoteJid\")
print(f\"IN {to}: {text}\", flush=True)
answer = ask_llm(text) # (3) think
send_whatsapp(to, answer) # (4) reply
print(f\"OUT {to}: {answer[:60]}...\", flush=True)
return {\"ok\": True}
The one line beginners always miss is (1) the loop guard. When the bridge sends a\nreply, that outgoing message fires another MESSAGES_UPSERT webhook with fromMe: true\nā without the guard, the bridge answers itself, forever. Rebuild:
docker compose up -d --build bridge
![\"The](\"https://development-wec.wiline.com/docs/assets/images/wa-bridge-logs-53cb3cf64a3d62d69794ee3a0bf3742b.png\")
\nFigure 3. v2 in action: a DM comes in, the model answers, the reply goes out.
It answers DMs now. But watch what happens next.
\nTail the logs with the bot live and you'll see it try to answer every group you're in,\nincluding promo/spam groups, and Evolution times out sending to them ā flooding errors:
\nIN 120363040812472138@g.us: *ALĆ CORREDORES š ... pechin.co/135449*
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='evolution-api', port=8080): Read timed out.
A companion account receives everything. Fix: answer DMs only ā groups end in\n@g.us:
to = key.get(\"remoteJid\", \"\")
if to.endswith(\"@g.us\") or \"broadcast\" in to: # DMs only
return {\"skipped\": \"not a DM\"}
Now the subtler wall. With groups filtered, a real DM comes in ā but the reply gets\nrejected:
\n{\"status\":400,\"error\":\"Bad Request\",\"response\":{\"message\":[
{\"exists\":false,\"jid\":\"123456789012345@lid\",\"name\":\"Test Contact\",\"number\":\"123456789012345@lid\"}]}}
![\"Evolution](\"https://development-wec.wiline.com/docs/assets/images/wa-lid-400-84cc0430abac7bdd5aa6d29cf1c100b5.png\")
\nFigure 4. The LID wall in the logs: the incoming DM arrives as @lid, and the reply is rejected with exists:false.
exists: false. This is WhatsApp's LID addressing (a 2025 privacy change): incoming\nDMs arrive with an @lid identifier, and you cannot send back to an @lid ā Evolution\nneeds the real @s.whatsapp.net phone JID. Evolution's contact store has both, though:
curl -s -X POST \"http://localhost:8080/chat/findContacts/wec-demo\" \\
-H \"apikey: $APIKEY\" -H \"Content-Type: application/json\" -d '{}' \\
| jq '[.[] | select(.pushName==\"Test Contact\")]'
[ { \"remoteJid\": \"123456789012345@lid\", \"pushName\": \"Test Contact\" },
{ \"remoteJid\": \"5511999999999@s.whatsapp.net\", \"pushName\": \"Test Contact\" } ]
![\"findContacts](\"https://development-wec.wiline.com/docs/assets/images/wa-findcontacts-275c01bee94ec0da990f2cdd0ec6acf2.png\")
\nFigure 5. Evolution stores both identities for the same person ā the @lid and the real phone JID. That is the key to resolving the reply address.
So we resolve the @lid to the phone number before replying. Add a resolver and use it:
def resolve_number(remote_jid: str, push_name: str):
if remote_jid.endswith(\"@s.whatsapp.net\"):
return remote_jid.split(\"@\")[0]
# LID: find the contact's real number by matching the name
contacts = requests.post(f\"{EVOLUTION_URL}/chat/findContacts/{INSTANCE}\",
headers={\"apikey\": EVOLUTION_APIKEY, \"Content-Type\": \"application/json\"},
json={}, timeout=30).json()
for c in contacts:
if c.get(\"pushName\") == push_name and c.get(\"remoteJid\", \"\").endswith(\"@s.whatsapp.net\"):
return c[\"remoteJid\"].split(\"@\")[0]
return None
In the handler, resolve before sending:
\n number = resolve_number(to, data.get(\"pushName\", \"\"))
if not number:
return {\"skipped\": \"unresolved lid\"}
...
send_whatsapp(number, answer)
Now SEND -> 201. (Matching by name is a heuristic ā two contacts sharing a name would\ncollide; production keeps a proper LIDāphone map. But it works, and it's the honest state\nof Evolution's LID support in v2.2.3.)
LLMs emit Markdown (**bold**, [text](url)), but WhatsApp has its own formatting:\nbold is *single asterisk*, and Markdown links don't render. So **Compute** shows up\nwith literal asterisks. Translate the model's output before sending, and append the RAG's\nsources as plain URLs:
import re
def to_whatsapp(md: str) -> str:
md = re.sub(r\"\\*\\*(.+?)\\*\\*\", r\"*\\1*\", md) # **bold** -> *bold*
md = re.sub(r\"\\[([^\\]]+)\\]\\((https?://[^)]+)\\)\", r\"\\1 (\\2)\", md) # [t](url) -> t (url)
md = re.sub(r\"(?m)^#{1,6}\\s*\", \"\", md) # drop # headings
return md
Verified against a real answer ā exactly what WhatsApp will render:
\nTo create a compute instance on WiLine Edge Cloud (WEC), follow these steps:
1. Log in to the WiLine Edge Cloud.
2. In the sidebar, click *Compute*.
3. Select *Instances*.
4. Click the \"Launch Virtual Machine\" wizard to begin the deployment process.
_Sources:_
https://wec.wiline.com/docs/cloud_portal/platform/compute/instances/compute_instance/
![\"The](\"https://development-wec.wiline.com/docs/assets/images/wa-formatted-output-766cfd620ca72685d8f37110da283d56.png\")
\nFigure 6. The model output after to_whatsapp(): single-asterisk bold, plain links, sources appended.
Everything so far uses the raw model, so a WEC question gets a general answer. To make it\nanswer from your documentation, swap ask_llm to call the RAG service from the\nevals series ā one function, nothing\nelse changes:
RAG_URL = \"http://<your-vm-ip>:8000/ask\"
def ask_llm(text: str) -> str:
r = requests.post(RAG_URL, json={\"question\": text}, timeout=120)
r.raise_for_status()
d = r.json()
answer = to_whatsapp(d[\"answer\"])
sources = d.get(\"sources\", [])[:2]
if sources:
answer += \"\\n\\n_Sources:_\\n\" + \"\\n\".join(sources)
return answer
That's the payoff of the bridge pattern: swap the brain, keep the plumbing. Now WhatsApp\nanswers WEC questions from the real docs, with sources:
\n![\"WhatsApp](\"https://development-wec.wiline.com/docs/assets/images/wa-bridge-reply-bf0eaa9a06721ee1396cbf44fd96ae3a.png\")
\nFigure 7. A real message in ā the bridge ā RAG over the WEC docs ā a grounded answer\ndelivered on WhatsApp, sources included.
You self-hosted a programmable WhatsApp gateway and wrote the bridge that turns it into\nan AI assistant ā webhook in, model out, reply back ā grounded in your own docs.
Here's the part other tutorials won't tell you. Even with SEND -> 201, replies to a\nLID-migrated account frequently arrive as \"Waiting for this message\" on the\nrecipient's phone ā WhatsApp accepted the ciphertext but no device can decrypt it. It's a\nknown Baileys companion-device limitation, and WhatsApp's LID rollout makes it worse.
What we observed, honestly:
\n\"Waiting for this message\" returns ā even though\nevery send still reports 201.So the bridge is correct end-to-end; the unofficial Baileys transport is the weak link.\nIf you're building this for real:
\nEvolution shines for outbound automation to numbers you control (notifications, alerts,\nflows). As a two-way AI assistant on a personal LID account, treat delivery as best-effort.
\npull access denied on atendai/evolution-api ā image moved to evoapicloud/evolution-api.connecting, re-init loop ā stale Baileys version; pin CONFIG_SESSION_PHONE_VERSION.fromMe loop guard.@g.us; answer DMs only.SEND -> 400 exists:false ...@lid ā resolve the LID to the phone JID via findContacts.**asterisks** in replies ā translate Markdown to WhatsApp formatting.cd ~/evolution-api && docker compose down # add -v to also wipe the DB/session
Unlink the device in WhatsApp ā Linked Devices if you're done.
\nYou built the reusable bridge pattern ā webhook ā resolve ā model ā reply. Point it at a\ndifferent brain, a different channel, or add tools. If you want an AI agent on WhatsApp\nwithout the Baileys caveats, the OpenClaw native WhatsApp\nchannel is the managed path; for docs-grounded\nanswers, the evals & observability series\nbuilds the RAG service this tutorial plugs into.
", "url": "https://development-wec.wiline.com/docs/tutorials/whatsapp-ai-assistant-evolution-api/", "title": "Build a WhatsApp AI assistant from scratch with Evolution and the WEC API", "summary": "Self-host a programmable WhatsApp gateway (Evolution API) and write the ~50-line bridge that turns it into an AI assistant ā webhook in, LLM out, reply back. Every gotcha is real: the image that moved publishers, the Baileys version loop, the loop guard, group spam, WhatsApp's LID addressing, and the delivery wall nobody warns you about.", "date_modified": "2026-07-21T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "whatsapp", "evolution-api", "docker", "inference" ] }, { "id": "https://development-wec.wiline.com/docs/tutorials/add-whatsapp-channel-openclaw/", "content_html": "![\"OpenClaw\"](\"https://development-wec.wiline.com/docs/assets/images/openclaw-wordmark-003352a1a7f02afc3bd877ae6f2dc175.png\")
+Telegram (Part 3) gave your agent a bot.\nWhatsApp gives it a phone line ā the app ~3 billion people already use, reachable with\nzero friction. One catch worth understanding up front: WhatsApp has no bot account, so\nOpenClaw links to a real number as a companion device (like WhatsApp Web) and the\nagent acts as that account. Every command and error below is from a real run.
\n\nAnyone messages the number; the gateway ā linked as a companion device ā runs it through\nyour model and replies from the number itself. No inbound ports, no bot account.
\n\nPrerequisites: Parts 1ā4 done (OpenClaw in ~/openclaw), a WhatsApp account on your\nphone, ~1 GB free disk. All commands run in ~/openclaw.
OpenClaw treats messaging apps as channels; WhatsApp is a first-class one:
\ndocker compose run --rm openclaw-cli channels add --channel whatsapp
Installed WhatsApp plugin
Added WhatsApp account \"default\".
requires plugin API >=2026.7.1? You're on an older OpenClaw ā upgrade first.The WhatsApp channel is a plugin gated to newer runtimes:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/wa-version-wall-db236509ca6e250b19147327473e0932.png\")
\nFigure 1. The version wall ā the WhatsApp plugin needs OpenClaw ā„2026.7.1.
Back up, pull, recreate, and confirm your existing channels survived (:latest does\nnot auto-update ā you must pull deliberately):
cp ~/.openclaw/openclaw.json ~/.openclaw/openclaw.json.bak-$(date +%Y%m%d)
docker compose pull openclaw-gateway
docker compose run --rm openclaw-cli --version # expect 2026.7.1+
docker compose up -d --force-recreate openclaw-gateway
docker compose run --rm openclaw-cli channels status # Telegram should still be connected
![\"The](\"https://development-wec.wiline.com/docs/assets/images/wa-version-check-b1b98be0f2990d46e5686130b2722d3b.png\")
\nFigure 2. After pulling, the image reports 2026.7.1 ā the requirement is met.
Then re-run the channels add above.
Check status and the channel is stopped with a cryptic error:
\n![\"WhatsApp](\"https://development-wec.wiline.com/docs/assets/images/wa-trust-error-7a8033b008450d82190f783b79998f71.png\")
\nFigure 3. stopped ā an untrusted plugin canāt use openKeyedStore, so the channel wonāt start.
That's a security feature: OpenClaw sandboxes third-party plugins, and an untrusted\none can't use openKeyedStore (the storage WhatsApp needs for its session). Trust it ā\ninclude any other non-bundled plugin already loading (here codex) so you don't lock it\nout ā and recreate:
docker compose run --rm openclaw-cli config set \\
--batch-json '[{\"path\":\"plugins.allow\",\"value\":[\"codex\",\"whatsapp\"]}]'
docker compose up -d --force-recreate openclaw-gateway
docker compose run --rm openclaw-cli channels status
![\"plugins.allow](\"https://development-wec.wiline.com/docs/assets/images/wa-trust-fix-503842c5815cf95dc4bde07dc3b4a6ee.png\")
\nFigure 4. Trusted ā plugins.allow now lists whatsapp (and codex).
The restart matters ā the gateway reads plugins.allow before it initializes the\nplugin. Now it's up:
![\"WhatsApp](\"https://development-wec.wiline.com/docs/assets/images/wa-connected-d4ec6a75486c355d24570487f62305c8.png\")
\nFigure 5. Live ā WhatsApp running, connected, healthy, and Telegram still up.
docker compose run --rm openclaw-cli channels login
Scan the terminal QR from WhatsApp ā Settings ā Linked Devices ā Link a Device.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/wa-qr-544d7332e052ce0469c77d7e60fec42a.png\")
\nFigure 6. Scan to link; ā
Linked after restart handles the code 515 reconnect.
code 515 restart is normalWhatsApp forces a reconnect right after pairing; OpenClaw handles it automatically. This\nis exactly where hand-rolled setups get stuck in a loop ā here it just works.
From another phone, message the number:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/wa-reply-08f4125cb353af3cff9a2418928be7ec.png\")
\nFigure 7. A question in, the agentās answer back ā sent from the number, as the account.
It replies ā but notice it answers from your number, as you. That's companion mode:\nthe agent is the account. Great for a personal assistant, and the reason for the warning\nbelow.
\nYour OpenClaw agent answers on WhatsApp ā natively, no bridge, no bot account.
It processes every DM and group the account receives (Listening for ... DM + all groups). For anything real: link a dedicated number (a cheap prepaid SIM ā virtual\nnumbers are often rejected), and set an allowlist before exposing it.
Park the channel when you're done testing (Telegram keeps running; your link stays saved):
\ndocker compose run --rm openclaw-cli config set \\
--batch-json '[{\"path\":\"channels.whatsapp.accounts.default.enabled\",\"value\":false}]'
docker compose up -d --force-recreate openclaw-gateway
openKeyedStore is only available for trusted plugins ā channel installs but stays\nstopped. Add the id to plugins.allow and recreate the gateway (Step 2). A live\nconfig change isn't enough once it has already failed to start.code 515 ā normal post-pairing restart. If it never settles, remove\nstale entries in WhatsApp ā Linked Devices and scan a fresh QR.Part 5 done ā your agent reaches Telegram and WhatsApp from a private, hardened box.
\ndocker compose run --rm openclaw-cli channels remove --channel whatsapp
docker compose restart openclaw-gateway
And unlink the device in WhatsApp ā Linked Devices if you're done.
", "url": "https://development-wec.wiline.com/docs/tutorials/add-whatsapp-channel-openclaw/", "title": "Add a WhatsApp channel to OpenClaw", "summary": "Put your self-hosted agent on WhatsApp. Add the channel, trust the plugin, scan a QR ā and understand why a companion link makes the agent act as your account. Every command and error from a real run.", "date_modified": "2026-07-20T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "openclaw", "whatsapp", "chatbot" ] }, { "id": "https://development-wec.wiline.com/docs/tutorials/deepeval-regression-testing-wiline-inference/", "content_html": "Our GPT-5.6 token-economics analysis ended with a challenge:\nbenchmark promises are a hypothesis, not a reason ā run the eval on your workload before you\nbelieve a \"fewer tokens per task\" claim. This tutorial is us taking our own advice.
\nWe take the RAG docs assistant from part 5,\nwrap it in a DeepEval regression suite ā containerized, judged by gemma4 on the WEC\nInference API, zero OpenAI dependency ā and then use that suite to answer a genuinely open\nquestion: should we swap the service's generation model? Qwen2.5-3B-Instruct (current) vs\ngemma4 (candidate), pass rate and tokens per task, measured head to head.
Spoiler: the eval saved us from a pointless migration. And along the way we hit four real\nproduction failures ā a disk-full death spiral, a startup race condition, a Docker Compose\nvariable trap that silently evaluated the wrong model, and the fix that makes that class of\nbug impossible. Every command, number, and error below is from a real run.
\n\nThree design decisions up front:
\ndocker compose run eval in both places. (Our first attempt was a\nvenv. It died to a full disk before installing anything; see Troubleshooting.)tests.csv embeds each\nreference fact inside an llm-rubric string ā one regex pulls it back out. Your eval data\noutlives your eval framework.rag-api answering on :8000, eval/tests.csv present).df -h / now. Ours read 100% and the story of why\nis in Troubleshooting.Never eval a service you haven't poked by hand first:
\ncurl -s -X POST localhost:8000/ask \\
-H \"Content-Type: application/json\" \\
-d '{\"question\": \"How do I create a compute instance on WEC?\"}' | python3 -m json.tool
![\"The](\"https://development-wec.wiline.com/docs/assets/images/deepeval-smoke-test-98290b8dda1f2a3f5e57a82fa43c2dfa.png\")
\nFigure 1. The system under test, alive: /ask answers from the WEC docs corpus and cites\nits sources. Never eval a service you haven't poked by hand first.
eval/Dockerfile:
FROM python:3.12-slim
WORKDIR /eval
RUN pip install --no-cache-dir deepeval==3.*
# Suite code is mounted at runtime so edits don't need a rebuild
CMD [\"deepeval\", \"test\", \"run\", \"test_rag_regression.py\"]
Add the service to docker-compose.yml inside the services: block (part 5's file ends\nwith a top-level volumes: section ā appending blindly puts your service in the wrong block;\ndocker compose config --quiet catches it):
eval:
build: ./eval
profiles: [\"eval\"]
env_file: .env
environment:
RAG_API_URL: http://rag-api:8000
volumes:
- ./eval:/eval
depends_on:
- rag-api
profiles: [\"eval\"] keeps it out of a normal docker compose up ā it only runs when you ask.\nBuild it:
docker compose --profile eval build eval
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/deepeval-eval-build-58810296ef048e28edf1a9f90735d2eb.png\")
\nFigure 2. The eval runner builds in ~42s: python:3.12-slim plus DeepEval 3.x. This exact\nimage is what CI runs later ā no venv, no drift.
DeepEval's LLM-as-judge metrics want an OpenAI key. We don't have one and don't want one.\neval/wec_judge.py:
\"\"\"Custom DeepEval judge backed by WiLine Inference (OpenAI-compatible).\"\"\"
import os
from openai import OpenAI
from deepeval.models.base_model import DeepEvalBaseLLM
WEC_BASE_URL = os.getenv(\"WEC_BASE_URL\", \"https://inference.wiline.com/v1\")
class WECJudge(DeepEvalBaseLLM):
def __init__(self, model: str | None = None):
self.model_name = model or os.getenv(\"JUDGE_MODEL\", \"gemma4\")
self.client = OpenAI(
base_url=WEC_BASE_URL,
api_key=os.environ[\"WEC_API_KEY\"],
)
def load_model(self):
return self.client
def generate(self, prompt: str) -> str:
response = self.client.chat.completions.create(
model=self.model_name,
messages=[{\"role\": \"user\", \"content\": prompt}],
temperature=0,
)
return response.choices[0].message.content
async def a_generate(self, prompt: str) -> str:
return self.generate(prompt)
def get_model_name(self):
return f\"WEC/{self.model_name}\"
Prove it answers from inside the compose network before building anything on it:
\ndocker compose --profile eval run --rm eval \\
python -c \"from wec_judge import WECJudge; j = WECJudge(); print('judge says:', j.generate('Reply with exactly: WEC judge online'))\"
judge says: WEC judge online
![\"Judge](\"https://development-wec.wiline.com/docs/assets/images/deepeval-judge-online-c5844beb37c759e19f698f4ea927ec8c.png\")
\nFigure 3. \"WEC judge online\" ā gemma4 answering through the container, on the same compose\nnetwork as the service it will judge. No OpenAI key anywhere in the stack.
An honest caveat we carry through the rest of this tutorial: in the model comparison later,\ngemma4 is both judge and contestant. The mitigation is structural ā the judge only compares\nanswer-vs-reference-fact, never \"which model wrote this\" ā and we watch its verdicts on the\nknown-hard cases for favoritism. (It showed none: every case it failed gemma4 on is a case it also fails Qwen on.)
\nYou can point DeepEval's LLM-judged metrics at any OpenAI-compatible endpoint. Your eval\nverdicts ā questions, answers, reasoning ā never leave your infrastructure.
Part 5's /ask returned answer and sources ā and threw away the token usage that came back\nwith every completion. For token economics we need it, and for the A/B we need the model to be\nswitchable. Three env vars and two response fields in app/main.py (volume-mounted; uvicorn\nhot-reloads, no rebuild):
# Generation model is configurable so the eval suite can A/B models.
# Defaults preserve the tutorial-5 setup.
GEN_MODEL = os.getenv(\"GEN_MODEL\", \"Qwen2.5-3B-Instruct\")
GEN_BASE_URL = os.getenv(\"GEN_BASE_URL\", \"https://inference.wiline.com/v1\")
GEN_API_KEY = os.getenv(\"GEN_API_KEY\", os.environ[\"WEC_API_KEY\"])
generate() now also returns resp.usage, and /ask reports both:
{
\"answer\": \"To view your billing statements ...\",
\"sources\": [\"...\"],
\"model\": \"Qwen2.5-3B-Instruct\",
\"usage\": {
\"prompt_tokens\": 1238,
\"completion_tokens\": 122,
\"total_tokens\": 1360
}
}
![\"/ask](\"https://development-wec.wiline.com/docs/assets/images/deepeval-ask-usage-b1a5b036ae1af43fa37d1c91c9d2f092.png\")
\nFigure 4. The new model and usage fields: 1,238 prompt + 122 completion tokens for one\nbilling question. This is the number the token-economics debate is actually about.
That usage block is the number the GPT-5.6 marketing is about ā measured on your workload,\nper request. Pass the variable through compose (GEN_MODEL: ${GEN_MODEL:-Qwen2.5-3B-Instruct}\nunder rag-api's environment:) and remember this line ā it comes back to bite us in\nTroubleshooting.
eval/test_rag_regression.py. It reads part 5's tests.csv, extracts the reference fact from\neach promptfoo rubric, asks the live service, logs token usage, and asserts a GEval correctness\nmetric judged by WECJudge:
\"\"\"DeepEval regression suite for the WEC docs RAG service.
Reuses the promptfoo golden set (tests.csv) from the previous tutorial:
the reference fact is parsed out of each llm-rubric string.
\"\"\"
import csv, json, os, re
import pytest
import requests
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from wec_judge import WECJudge
RAG_API_URL = os.getenv(\"RAG_API_URL\", \"http://rag-api:8000\")
USAGE_LOG = os.getenv(\"USAGE_LOG\", \"usage_log.jsonl\")
MAX_CASES = int(os.getenv(\"MAX_CASES\", \"0\")) # 0 = all
FACT_RE = re.compile(r'Reference fact from the official docs: \"(.*?)\"')
def load_cases():
with open(\"tests.csv\", newline=\"\") as f:
for row in csv.DictReader(f):
m = FACT_RE.search(row[\"__expected\"])
yield row[\"question\"], (m.group(1) if m else \"\")
CASES = list(load_cases())
if MAX_CASES:
CASES = CASES[:MAX_CASES]
judge = WECJudge()
correctness = GEval(
name=\"Correctness\",
criteria=(
\"The actual output must correctly address the input question and must not \"
\"contradict the reference fact given as expected output. Different wording \"
\"is fine. Fail only if the answer is wrong, contradicts the reference fact, \"
\"or fails to answer the question.\"
),
evaluation_params=[
LLMTestCaseParams.INPUT,
LLMTestCaseParams.ACTUAL_OUTPUT,
LLMTestCaseParams.EXPECTED_OUTPUT,
],
model=judge,
threshold=0.5,
)
@pytest.mark.parametrize(
\"question,reference\", CASES, ids=[q[:50] for q, _ in CASES]
)
def test_rag_regression(question, reference):
r = requests.post(f\"{RAG_API_URL}/ask\", json={\"question\": question}, timeout=120)
r.raise_for_status()
data = r.json()
expect = os.getenv(\"EXPECT_MODEL\")
assert not expect or data[\"model\"] == expect, (
f\"suite expected {expect} but service is running {data['model']}\"
)
with open(USAGE_LOG, \"a\") as f:
f.write(json.dumps({
\"model\": data.get(\"model\"),
\"question\": question,
**data.get(\"usage\", {}),
}) + \"\\n\")
assert_test(
LLMTestCase(
input=question,
actual_output=data[\"answer\"],
expected_output=reference,
),
[correctness],
)
(That EXPECT_MODEL assertion wasn't in the first version. It exists because of a bug you'll\nmeet in a moment.)
First run, capped at 5 cases to fail fast:
\ndocker compose --profile eval run --rm \\
-e MAX_CASES=5 -e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
eval
4 of 5 passed in 2:37 ā and both the passes and the failure are worth reading. The\nfailure (\"What is the starting point for billing?\", score 0.3) isn't a service bug: the\nanswer was factually fine, but the golden label is a section title (\"How Billing Works\")\nand the judge wanted it named. Remember this case ā it flickers between pass and fail all\ntutorial long, and that flicker becomes a finding. Meanwhile on \"total number of events\"\nthe judge scored a 0.6 borderline pass with exactly the right reasoning (the service\ndescribed the feature instead of giving steps). Nuanced verdicts from a self-hosted judge ā\nthe thing people assume you need GPT-4-class models for.
\n![\"First](\"https://development-wec.wiline.com/docs/assets/images/deepeval-first-run-cabfff276f20b525d0c7c20bb52c1a6a.png\")
\nFigure 5. First run: 4 of 5 passed in 2:37, every verdict with written reasoning from\nthe gemma4 judge ā including the one failure, a strict verdict on a section-title label\nrather than a service bug.
Note DeepEval's summary line: token cost: None. It can't price a self-hosted judge ā which is\nexactly why the suite writes its own usage_log.jsonl.
![\"Per-question](\"https://development-wec.wiline.com/docs/assets/images/deepeval-usage-log-65042f9bd05652b9ff3814cd52b37270.png\")
\nFigure 6. The suiteās own accounting: one JSON line per question ā model, prompt, completion,\ntotal ā the raw data every comparison in this tutorial is built from.
You have a regression suite grading a live RAG service ā reusing last tutorial's golden set\nunchanged, judged by a self-hosted model, with per-question token accounting the framework\nitself doesn't provide.
The question the suite exists to answer: our service generates with Qwen2.5-3B-Instruct ā\nshould it? gemma4 is newer, well-regarded, already in the WEC catalog. Feels like an upgrade.\nFeels. Let's measure.
\nBaseline, all 16 cases:
\ndocker compose --profile eval run --rm \\
-e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
-e USAGE_LOG=usage_qwen.jsonl \\
eval
14/16 (87.5%) in 8:50. The two failures are label artifacts, not RAG bugs: the golden facts\nfrom part 5 are section titles (\"How Billing Works\"), and the judge dinged answers that were\nfactually fine but didn't echo the title; and mp3 vs the label audio/mpeg ā the same fact at\ntwo abstraction levels, ruled a contradiction. Read your failures before trusting your pass\nrate. We leave both in: they're honest label debt, and they become our judge-favoritism canary\nin the A/B.
![\"Qwen](\"https://development-wec.wiline.com/docs/assets/images/deepeval-qwen-baseline-5b61265199e0d615241bbbfdb30d7a81.png\")
\nFigure 7. The baseline: 87.5% in 8:50. Both failures are label artifacts, not RAG bugs ā\nread your failures before trusting your pass rate.
Challenger ā same suite, one env var (prefix it on every compose command; Troubleshooting\nexplains why in blood):
\nGEN_MODEL=gemma4 docker compose up -d rag-api
until curl -sf localhost:8000/health > /dev/null; do echo \"waiting...\"; sleep 3; done
curl -s -X POST localhost:8000/ask -H \"Content-Type: application/json\" -d '{\"question\": \"ping\"}' \\
| python3 -c \"import json,sys; print('>>> model is:', json.load(sys.stdin)['model'])\"
GEN_MODEL=gemma4 docker compose --profile eval run --rm \\
-e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
-e USAGE_LOG=usage_gemma4.jsonl \\
-e EXPECT_MODEL=gemma4 \\
eval
And verify the log, because run labels lie:
\nsort <(cat eval/usage_gemma4.jsonl | python3 -c \"import json,sys; [print(json.loads(l)['model']) for l in sys.stdin]\") | uniq -c
# 16 gemma4
![\"Switching](\"https://development-wec.wiline.com/docs/assets/images/deepeval-gemma4-run-6a7c64183ec8b68e6d5e5ff76dd0924a.png\")
\nFigure 8. Switching the challenger in: GEN_MODEL=gemma4, wait for /health, then confirm\n>>> model is: gemma4 from the response itself ā before spending a single eval minute.
![\"gemma4](\"https://development-wec.wiline.com/docs/assets/images/deepeval-gemma4-verified-8d7291e9dfcda2f337f83ca7e4988071.png\")
\nFigure 9. The gemma4 run: 81.25% in ~13 minutes ā and its three failures are the same\nlabel-artifact cases Qwen flickers on. No favoritism from the gemma4 judge toward the gemma4\ncontestant.
One script over the usage logs:
\n| run | pass rate | avg prompt tokens | avg completion tokens | total tokens |
|---|---|---|---|---|
| Qwen2.5-3B (run 1) | 87.5% | 1,126 | 75 | 19,225 |
| Qwen2.5-3B (run 2) | 81.25% | 1,126 | 75 | 19,225 |
| gemma4 | 81.25% | 1,199 | 415 | 25,834 |
![\"Token](\"https://development-wec.wiline.com/docs/assets/images/deepeval-verdict-table-fc9c10d332e08f09d63c768e48c1b23c.png\")
\nFigure 10. The verdict in five lines: same pass rate, 5.5Ć the completion tokens. The eval\njust prevented a pointless migration.
Two findings, one expected and one not:
\nThe verdict: don't switch. Same effective quality, same failure cases ā but gemma4 uses\n5.5Ć the completion tokens (415 vs 75 per answer) and runs ~45% slower per question. On a\nRAG workload where the ~1,100-token retrieved context dominates the prompt, the completion\ncolumn is the whole cost difference. \"Newer model\" lost to \"measure it\" in one afternoon. This\nis precisely the eval the GPT-5.6 news post said to run ā and it's the same eval you'd run\nagainst GPT-5.6 itself: change GEN_MODEL, GEN_BASE_URL, and GEN_API_KEY, nothing else.
The judge has a noise floor. Look at the two Qwen rows: identical token counts ā\n19,225 in both runs, because temperature-0 generation is deterministic ā yet 87.5% vs 81.25%.\nThe answers didn't change; only the judge's verdicts did (one borderline 0.6 became a 0.3).\nThat cleanly isolates judge nondeterminism from model behavior, and it sets your CI threshold:\na gate at \"ā„ 87.5%\" would flake. Ours tolerates one flipped verdict and fails on two or more\nreal regressions.
\n![\"Qwen](\"https://development-wec.wiline.com/docs/assets/images/deepeval-qwen-run2-48156c484cd782d1e346e0938b1080b9.png\")
\nFigure 11. Qwen run 2: the same 16 deterministic answers, 81.25% this time ā the judge, not\nthe model, is where the variance lives.
You can settle a model-swap debate with measured data ā pass rate and tokens per task ā and\nyou know your judge's noise floor, so you can set a CI threshold that won't flake.
The payoff of containerizing everything: the CI job is the command you've been running all\nalong. .github/workflows/eval.yml:
name: rag-regression-eval
on:
pull_request:
paths: [\"app/**\", \"eval/**\"]
workflow_dispatch:
jobs:
eval:
runs-on: [self-hosted, wec] # runner with access to inference.wiline.com
steps:
- uses: actions/checkout@v4
- name: Run regression suite
env:
WEC_API_KEY: ${{ secrets.WEC_API_KEY }}
run: |
docker compose up -d rag-api
docker compose --profile eval run --rm \\
-e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
-e EXPECT_MODEL=Qwen2.5-3B-Instruct \\
eval
- name: Teardown
if: always()
run: docker compose down
Any PR that touches the app or the eval must survive 16 judged questions against the live\nservice ā with the model identity asserted ā before it merges.
\nYour eval is a CI gate. The container you ran locally is byte-for-byte what CI runs ā no\n\"works on my machine\" gap between development and the pipeline.
[Errno 28] No space left on device ā and it came backāOur first pip install deepeval died: disk 100% full, 58G of 58G. We freed 2 GB of journal\nlogs and build cacheā¦ and it was full again minutes later. The culprit:
7.9G /var/lib/docker/containers/10281d.../10281d...-json.log
langfuse-clickhouse-1 was in an error loop, spraying stack traces into a Docker json-log with\nno rotation configured ā a death spiral, because a full disk causes more errors causes more\nlog. It had been \"unhealthy\" for 8 days. The fix, in order (order matters ā truncating while\nthe container runs just refills):
docker stop langfuse-clickhouse-1
sudo sh -c \"truncate -s 0 /var/lib/docker/containers/10281d*/*-json.log\"
Then make it permanent ā log rotation in the compose override, applied to every service:
\nx-logging: &default-logging
logging:
driver: json-file
options:
max-size: \"50m\"
max-file: \"3\"
services:
clickhouse:
<<: *default-logging
# ... every other service
docker compose up -d to recreate. ClickHouse went healthy for the first time in 8 days.\nDocker's json-file driver does not rotate by default; on a box running LLM infrastructure\nthat logs enthusiastically, unbounded container logs are a time bomb.
Connection refused, then 3 passāChanging GEN_MODEL recreates rag-api, and the app takes ~3 minutes to boot (Chroma +\nembedding model load). Our suite started hammering /ask immediately: 13 straight\nconnection-refused failures, then the API came alive mid-run and the last 3 passed. Flaky in\nthe worst way ā the failure depends on wall-clock timing.
Fix it in the suite, not in human discipline ā eval/conftest.py:
\"\"\"Refuse to start the suite until the RAG API is healthy.\"\"\"
import os, time
import requests
RAG_API_URL = os.getenv(\"RAG_API_URL\", \"http://rag-api:8000\")
READY_TIMEOUT = int(os.getenv(\"READY_TIMEOUT\", \"300\"))
def pytest_configure(config):
deadline = time.time() + READY_TIMEOUT
while time.time() < deadline:
try:
requests.get(f\"{RAG_API_URL}/health\", timeout=3).raise_for_status()
return
except requests.RequestException:
time.sleep(3)
raise RuntimeError(f\"rag-api not healthy after {READY_TIMEOUT}s ā aborting eval\")
The nastiest one, because nothing errored. ${GEN_MODEL:-Qwenā¦} in compose is interpolated\nfrom your shell at every invocation. We started the service with\nGEN_MODEL=gemma4 docker compose up -d ā correct. Then ran the eval without the prefix.\nCompose resolved the variable back to the Qwen default, saw the running container as config\ndrift, and ā via depends_on ā silently recreated rag-api with Qwen. The run completed,\nthe summary said \"gemma4\" in every verdict (the judge was gemma4), and every answer had been\nwritten by Qwen.
We only caught it because the suite logs the model from the response:
\nsort <(... usage_gemma4.jsonl ...) | uniq -c
# 16 Qwen2.5-3B-Instruct ā the \"gemma4\" run
Never trust the label on a run; verify from data the system under test reports about itself.\nThe permanent fix is the EXPECT_MODEL assertion in the suite (Step 4). Proof it works ā the\ndeliberate-failure test:
FAILED ... AssertionError: suite expected gemma4 but service is running Qwen2.5-3B-Instruct
1 failed in 2.44s
Under three seconds, before spending a single judge call ā versus the 10+ minutes each mislabeled run\ncost us.
\n![\"EXPECT_MODEL](\"https://development-wec.wiline.com/docs/assets/images/deepeval-expect-model-ccc34be3d936ecfd93823c565073b405.png\")
\nFigure 12. The guard in action: wrong model ā loud failure in under three seconds, before a\nsingle judge call is spent. Compare with the 10+ minutes each silently mislabeled run cost us.
sleep 5 is not a readiness checkāBonus small one: after recreating the container we piped /ask output into a JSON parser 5\nseconds later and got json.decoder.JSONDecodeError: Expecting value: line 1 column 1 ā the\nserver wasn't up, curl returned empty, the parser choked on nothing. Poll /health in a loop;\nnever sleep a guessed number.
gemma4 + GEval): verdicts with reasoning, no OpenAI keyEXPECT_MODEL guard that turns silently-evaluated-the-wrong-model into an under-three-second loud failureThe suite judges answers. Part 4's Langfuse traces know why ā which chunks were retrieved,\nwhat the generation span cost, where latency hides. Wiring eval failures back to their traces\n(DeepEval test ID ā Langfuse trace ID) turns \"test failed\" into \"here's the retrieval that\ncaused it.\" That's the component-level debugging story, and it's where this series goes next.
\n![\"Promptfoo\"](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-avatar-31b32c657d72dac0a7edbd77c24ec717.jpg\"){kind=avatar}
+Everything in this series was building to this. You can prove a model works\n(part 1), make its output machine-reliable\n(part 2), generate real test data\n(part 3), and observe production\n(part 4). Now we spend all four skills at once on\nthe pattern behind almost every serious LLM product: RAG ā retrieval-augmented generation.
\nWe'll build a docs assistant: a containerized HTTP service that answers customer questions\nfrom WEC's own documentation. Not a notebook ā a service, Docker-first, the shape you'd actually\ndeploy. And because this series doesn't do happy-path demos: along the way our RAG hallucinates a\nGPU price, we root-cause it to our own scraper, fix it, and pin the fix with a regression\ntest. Every command, number, and error below is from a real run.
\n\nOne architecture decision up front: we split the work. Embeddings run locally (a small ONNX\nmodel ā fast, free, no GPU needed) and generation runs on WEC. This is a completely standard\nproduction split: retrieval is cheap and latency-sensitive, so keeping it next to your vector\nstore saves a network round-trip per query ā while generation is where the big model earns its\nkeep.
\nThe WEC catalog now ships bge-m3 (1024-dim, multilingual, batch input) on\n/v1/embeddings:
curl -s https://inference.wiline.com/v1/embeddings \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"bge-m3\",\"input\":[\"first chunk\",\"second chunk\"]}' | jq '.data | length'
Swapping this tutorial over is a two-function change ā replace the fastembed calls in\ningest.py and main.py with the same OpenAI client pointed at /v1/embeddings\n(wec.embeddings.create(model=\"bge-m3\", input=chunks)), and re-ingest (the vector dimension\nchanges from 384 to 1024, so drop and rebuild the collection). We keep the local path below\nbecause it's free, offline, and every number in this guide was measured with it ā but both are\nproduction-legitimate; pick per your latency and ops preferences.
df -h /. Our box was at 99% and survived, but\ntight disk is the #1 silent killer of Docker builds.Real RAG features ship as services, so we start as one: a FastAPI app in a container, code\nbind-mounted so uvicorn --reload picks up every edit ā no rebuild per change.
mkdir -p ~/rag-service/app && cd ~/rag-service
cat > requirements.txt <<'EOF'
fastapi==0.115.6
uvicorn==0.34.0
chromadb==0.5.23
fastembed==0.4.2
openai==1.59.7
langfuse
requests==2.32.3
beautifulsoup4==4.12.3
EOF
cat > Dockerfile <<'EOF'
FROM python:3.12-slim
WORKDIR /srv
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ app/
EXPOSE 8000
CMD [\"uvicorn\", \"app.main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\", \"--reload\"]
EOF
cat > docker-compose.yml <<'EOF'
services:
rag-api:
build: .
ports:
- \"8000:8000\"
volumes:
- ./app:/srv/app # live code - edit on host, uvicorn reloads
- chroma-data:/data # persisted vector index
- model-cache:/root/.cache # embedding model cache (survives rebuilds)
environment:
ANONYMIZED_TELEMETRY: \"False\"
WEC_API_KEY: ${WEC_API_KEY}
LANGFUSE_PUBLIC_KEY: ${LANGFUSE_PUBLIC_KEY}
LANGFUSE_SECRET_KEY: ${LANGFUSE_SECRET_KEY}
LANGFUSE_HOST: ${LANGFUSE_HOST}
volumes:
chroma-data:
model-cache:
EOF
Secrets live in an .env file next to the compose (compose loads it automatically ā keys never\nenter the YAML). Keep it out of version control from the start:
cat > .env <<'EOF'
WEC_API_KEY=sk-your-wec-key
LANGFUSE_PUBLIC_KEY=pk-lf-ā¦
LANGFUSE_SECRET_KEY=sk-lf-ā¦
LANGFUSE_HOST=http://<your-langfuse-host>:3000
EOF
echo \".env\" >> .gitignore
A minimal app so the container has something to serve:
\ncat > app/main.py <<'EOF'
from fastapi import FastAPI
app = FastAPI(title=\"WEC docs RAG\")
@app.get(\"/health\")
def health():
return {\"status\": \"ok\"}
EOF
docker compose up -d --build
curl -s localhost:8000/health
{\"status\":\"ok\"}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/rag-deploy-health-850c74e1ed3eff7a20368db371456f79.png\")
\nFigure 1. One command each: the container up, /health answering, and the disk check\n(ours reads 99% ā this box lives dangerously). The build takes ~90s and a ~700 MB image.
In the real world, the knowledge base you'll want your assistant to answer from usually isn't a\ntidy folder of markdown ā it's a live website: product docs, a wiki, a help center. So the\ncorpus-acquisition skill worth learning is real scraping, not file loading.
\nFor this tutorial we scrape WEC's own documentation site. It's built with Docusaurus, and\nDocusaurus (like most site generators) publishes a sitemap.xml ā which means you can discover\nevery page programmatically instead of guessing URLs. The same approach works on any site that\nships a sitemap:
curl -s https://wec.wiline.com/docs/sitemap.xml | grep -c \"<loc>\" # 109 URLs on the whole site
We filter to the product docs (/docs/cloud_portal/), pull each page, and keep only the\n<article> element ā Docusaurus wraps the actual content in it, so nav/sidebar/footer never reach\nthe index:
cat > app/ingest.py <<'EOF'
\"\"\"Scrape the WEC docs site -> chunk -> embed locally -> Chroma.\"\"\"
import re, requests, chromadb
from bs4 import BeautifulSoup
from fastembed import TextEmbedding
SITEMAP = \"https://wec.wiline.com/docs/sitemap.xml\"
FILTER = \"/docs/cloud_portal/\" # product docs only
CHUNK, OVERLAP = 700, 100 # chars
def discover():
xml = requests.get(SITEMAP, timeout=30).text
urls = re.findall(r\"<loc>([^<]+)</loc>\", xml)
return [u for u in urls if FILTER in u]
def scrape(url):
html = requests.get(url, timeout=30).text
art = BeautifulSoup(html, \"html.parser\").find(\"article\") # Docusaurus main content
return art.get_text(\"\\n\", strip=True) if art else \"\"
def chunk(text):
out, i = [], 0
while i < len(text):
out.append(text[i:i + CHUNK])
i += CHUNK - OVERLAP
return out
if __name__ == \"__main__\":
urls = discover()
print(f\"sitemap -> {len(urls)} product-doc pages\")
docs, metas = [], []
for u in urls:
for c in chunk(scrape(u)):
docs.append(c)
metas.append({\"url\": u})
print(f\"scraped -> {len(docs)} chunks\")
embedder = TextEmbedding(\"BAAI/bge-small-en-v1.5\") # local, ~66MB ONNX, no GPU needed
vectors = [v.tolist() for v in embedder.embed(docs)]
print(f\"embedded -> {len(vectors)} vectors (dim {len(vectors[0])})\")
db = chromadb.PersistentClient(path=\"/data\")
col = db.get_or_create_collection(\"wec-docs\")
col.add(ids=[str(i) for i in range(len(docs))],
documents=docs, metadatas=metas, embeddings=vectors)
print(f\"indexed -> collection 'wec-docs' now has {col.count()} chunks\")
EOF
docker compose exec rag-api python -m app.ingest
sitemap -> 37 product-doc pages
scraped -> 281 chunks
model_optimized.onnx: 100%|āāāāāāāāāāāā| 66.5M/66.5M [00:03<00:00, 16.9MB/s]
embedded -> 281 vectors (dim 384)
indexed -> collection 'wec-docs' now has 281 chunks
![\"Ingest](\"https://development-wec.wiline.com/docs/assets/images/rag-ingest-2a1ffde1c98f1968bbf60532fa0a0a00.png\")
\nFigure 2. The whole product-doc corpus indexed in under a minute. The embedding model caches in\nthe model-cache volume, so it downloads exactly once.
You can turn any documentation website into a queryable vector index ā sitemap discovery, content\nextraction, chunking, local embeddings.
get_text(\"\\n\", ā¦)Our first version used get_text(\" \") ā a single space as the separator. It looked harmless and\ncaused a genuine hallucination you'll see in Step 4. Keep the newline; we'll come back to this.\n(Want to watch the bug happen on your own box? Change \"\\n\" back to \" \" in scrape(), re-run\nthe ingest, and ask the Step-4 question ā then put the newline back and re-ingest.)
You may see Failed to send telemetry event ClientStartEvent: capture() takes 1 positional argumentā¦ ā a known chromadb/posthog version clash. It's harmless (your data is fine), and the\nANONYMIZED_TELEMETRY: \"False\" env in the compose silences it.
/ask endpoint: retrieve, generate, traceāNow the service itself. Three design choices worth stating:
\nsources is deterministic.langfuse.openai (part 4's drop-in), and each stage gets an\n@observe span ā so every request produces a ask ā retrieve ā generate tree in Langfuse.timeout=60 on the client. Part 4's lesson: no default timeout means a stalled backend hangs\nyour service silently.cat > app/main.py <<'EOF'
import os, chromadb
from fastapi import FastAPI
from pydantic import BaseModel
from fastembed import TextEmbedding
from langfuse import observe, get_client
from langfuse.openai import openai # auto-traces the WEC call
app = FastAPI(title=\"WEC docs RAG\")
db = chromadb.PersistentClient(path=\"/data\")
col = db.get_or_create_collection(\"wec-docs\")
embedder = TextEmbedding(\"BAAI/bge-small-en-v1.5\")
wec = openai.OpenAI(
base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"],
timeout=60,
)
class Ask(BaseModel):
question: str
@observe() # span: retrieve
def retrieve(question: str, k: int = 4):
qv = list(embedder.embed([question]))[0].tolist()
res = col.query(query_embeddings=[qv], n_results=k)
chunks = res[\"documents\"][0]
sources = sorted({m[\"url\"] for m in res[\"metadatas\"][0]})
return chunks, sources
@observe() # span: generate (WEC call nested inside)
def generate(question: str, chunks: list[str]) -> str:
context = \"\\n---\\n\".join(chunks)
resp = wec.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\", # small on purpose: cheap + fast per query,
# and its failures teach (see Step 5)
name=\"rag-generate\",
messages=[{\"role\": \"user\", \"content\":
\"Answer the question using ONLY the context below. \"
\"If the context doesn't contain the answer, say so. \"
\"When quoting a price or number, copy it VERBATIM with its unit and the item \"
\"it belongs to; never combine numbers from different lines and never compute \"
\"new numbers from examples.\\n\\n\"
f\"Context:\\n{context}\\n\\nQuestion: {question}\"}],
)
return resp.choices[0].message.content
@app.post(\"/ask\")
@observe() # root span: the whole request
def ask(body: Ask):
chunks, sources = retrieve(body.question)
answer = generate(body.question, chunks)
get_client().flush()
return {\"answer\": answer, \"sources\": sources}
@app.get(\"/health\")
def health():
return {\"status\": \"ok\"}
EOF
docker compose up -d --build
Ask it a real customer question:
\ncurl -s localhost:8000/ask -H \"Content-Type: application/json\" \\
-d '{\"question\": \"How do I create an API key for the Inference API?\"}' | python3 -m json.tool
{
\"answer\": \"To create an API key for the Inference API, follow these steps:\\n1. Log in to the WiLine Edge Cloud.\\n2. Expand the \\\"Inference\\\" section in the left sidebar.\\n3. Click on the \\\"API Keys\\\" tab.\\n4. Click the \\\"+ Create Key\\\" button located above the keys table. ...\",
\"sources\": [
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/api/wiline-edge-cloud-inference-api/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/api_keys/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/examples/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/models_hub/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/overview/\"
]
}
![\"First](\"https://development-wec.wiline.com/docs/assets/images/rag-first-ask-301b6dfc7a64a463e0eb1e6d93b53837.png\")
\nFigure 3. First try: the exact API-key flow from the docs, with the right page cited. Scrape ā\nembed ā retrieve ā generate, end to end.
Open Langfuse ā Tracing and you'll find the request as a span tree ā ask ā retrieve ā generate ā rag-generate ā with per-step latency and tokens. When an answer is bad, this is how you\ntell bad retrieval from bad generation (part 4's skill, now on a real app).
![\"The](\"https://development-wec.wiline.com/docs/assets/images/rag-trace-tree-769e5224814989dedc4896f9e4e1873f.png\")
\nFigure 4. Every /ask becomes a tree: retrieval and generation each have an address.
You have a traced, containerized RAG service answering from live-scraped documentation.
A good RAG must refuse what the docs don't cover. Probe it:
\ncurl -s localhost:8000/ask -H \"Content-Type: application/json\" \\
-d '{\"question\": \"Does WEC offer a free GPU tier for students?\"}' | python3 -m json.tool
Our first version (the get_text(\" \") scraper) answered:
\"No, WEC does not offer a free GPU tier specifically for students. The pricing for
GPU hourly is listed as $0.66/GPU-hr with a 25% off annual prepay option at
$0.10/GB-mo for block storage. There is no mention of a free GPU tier in the
provided information.\"
The refusal is right. But read the pricing sentence again: \"$0.66/GPU-hr with a 25% off annual\nprepay option at $0.10/GB-mo for block storage\" ā that's three different facts welded into one\nfalse statement, gluing a storage rate onto the GPU prepay option. Check the actual pricing page\nand you find they're simply three adjacent items:
\nThe model stitched a storage price onto \"GPU\". Why? Look at the chunk we fed it. Our\nspace-separated scrape had flattened the pricing cards into one undifferentiated line:
\n... $1.44 /mo Starting at $0.66 /GPU-hr GPU hourly 25 % off Annual prepay $0.10 /GB-mo Block storage ...
The hallucination was manufactured by our own ingestion. The model conflated prices because we\ndestroyed the structure before embedding it. This is the most classic RAG failure there is ā\ngarbage in, garbage out ā and you only catch it by probing with questions and reading the\nretrieved chunks.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/rag-hallucination-18e9fb70cb217a347f31057d36b774e8.png\")
\nFigure 5. The bug in the wild: correct refusal, garbled pricing ā a storage rate glued onto the\nGPU prepay option. The sources look perfectly plausible. This is why \"it cites sources\" is not the\nsame as \"it's grounded.\"
Two fixes, both already in the code above:
\nget_text(\"\\n\", strip=True) keeps line boundaries, so price\ncards stay separate lines instead of one soup.After re-ingesting with the fix, the storage-price conflation is gone. Residual ambiguity remains ā\nthe pricing strip still contains fragments like \"$1.44/mo\" (a compute-instance price) near the GPU\nline, and truly fixing that needs table-aware extraction (see What's next). Which is exactly why\nthe next step exists: pin the fixed bug so it can never quietly return.
\nThis step took four rounds of fixes to get an honest number ā and the journey from 39% to 87%\nis the best lesson in the series, because almost nothing we fixed along the way was the RAG itself.
\nWe reuse part 3's skill ā generate a QA dataset from the corpus itself ā and grade the live\nHTTP endpoint with Promptfoo. The first generator version asked for one question per page with a\nkey_fact the answer must contain, and got 37/37 pairs back. Some labels already smelled ā a vague\n\"All paid\", menu paths instead of answers ā but let's run it naively first and let the failures\nteach us. Convert to a Promptfoo dataset with plain substring assertions (icontains: + the key\nfact) and point it at the running service. The provider is HTTP ā we grade the real API,\nexactly what a CI gate would do:
mkdir -p eval
{ echo 'question,__expected'; jq -r '[.question, (\"icontains:\" + .key_fact)] | @csv' app/qa.jsonl; } > eval/tests.csv
cat > eval/promptfooconfig.yaml <<'EOF'
description: \"RAG service eval ā QA set generated from the scraped docs + regression\"
providers:
- id: https
config:
url: http://localhost:8000/ask
method: POST
headers:
Content-Type: application/json
body:
question: \"{{question}}\"
transformResponse: json.answer
tests:
- file://tests.csv
# regression: the storage-price-as-GPU-price conflation from Step 4
- vars:
question: \"What is the pricing model for GPU hourly usage in Edge Cloud?\"
assert:
- type: not-icontains
value: \"0.10\"
- type: not-icontains
value: \"GB-mo\"
EOF
cd eval && npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
Results:
ā 15 passed (39.47%)
ā 23 failed (60.53%)
Duration: 40s (concurrency: 4)
That 39.5% is what we measured during development, while the Step-4 bug was still live. Re-run\nthe same naive eval against the finished service and it lands much higher ā 27/38:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/rag-eval-icontains-284e4ea4fa5a8f2dec99a5079c84c4f7.png\")
\nFigure 6. The same naive eval against the finished service: 71.05%. Better service, same\nnoisy labels ā the 30-point gap between these two runs is all service quality, and both numbers\nmean little until you read the rows.
Read the failures before you panic. From the development run's 23 failures, most are not RAG\nfailures:
\n| Failure | What actually happened |
|---|---|
| \"Balance due\" answer: \"the amount that remains dueā¦ $0.00\" | Correct answer; the label demanded the phrase All paid ā bad label |
| Payment Information: gives the exact right menu path | Correct; label phrasing doesn't substring-match ā brittle assertion |
| GPU pricing: \"$0.10 per GB-month\" | Real failure ā the Step-4 conflation, reproducing systematically |
| TTS default format: \"not specified in the context\" | Real retrieval miss ā the chunk with the default wasn't retrieved |
Substring matching (icontains) was perfect for part 2's single JSON field and is too brittle\nfor long-form RAG answers. The obvious upgrade is part 4's skill: an LLM judge grading\nsemantic consistency (llm-rubric: assertions + a judge model in defaultTest). We did exactly\nthat ā same 37 labels, judge instead of substrings ā and got:
Results:
ā 17 passed (44.74%)
ā 21 failed (55.26%)
Duration: 8m 21s (concurrency: 4)
39.5% ā 44.7%. Barely moved ā because the judge faithfully enforces bad labels. Rows with\ngarbage key_facts still fail: the reference is wrong, not the answer. A better grader cannot\nrescue a bad dataset. The bottleneck was never the assertion type.
So fix the data (part 3's discipline, now enforced by code). Generator v2 demands the label be\nthe answer ā not a menu path, not a section name ā and adds a curation gate: the key_fact\nmust exist verbatim in the source text, or the row is rejected on the spot:
cat > app/gen_qa.py <<'EOF'
\"\"\"Generate a QA eval set from the indexed docs ā v2, curated labels.\"\"\"
import os, json, chromadb
from openai import OpenAI
wec = OpenAI(base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"], timeout=60)
db = chromadb.PersistentClient(path=\"/data\")
col = db.get_or_create_collection(\"wec-docs\")
data = col.get(include=[\"documents\", \"metadatas\"])
pages = {}
for doc, meta in zip(data[\"documents\"], data[\"metadatas\"]):
pages.setdefault(meta[\"url\"], []).append(doc)
kept, skipped = 0, 0
with open(\"/srv/app/qa.jsonl\", \"w\") as f:
for url, chunks in sorted(pages.items()):
ctx = \"\\n\".join(chunks[:2])[:2000]
resp = wec.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\", temperature=0.3,
messages=[{\"role\": \"user\", \"content\":
\"From this documentation excerpt, write ONE question a customer would ask, \"
\"and the fact that ANSWERS it. Return ONLY JSON with keys:\\n\"
\"question (string), key_fact (string).\\n\"
\"Rules for key_fact: it must be THE ANSWER to the question (not a menu path, \"
\"not a section name), a short phrase copied VERBATIM from the excerpt, \"
\"max 8 words.\\n\\n\"
f\"Excerpt:\\n{ctx}\"}],
)
raw = resp.choices[0].message.content
try:
start, end = raw.index(\"{\"), raw.rindex(\"}\") + 1
row = json.loads(raw[start:end])
# curation gate: the label must literally exist in the source text
if row[\"key_fact\"].lower().strip() not in ctx.lower():
print(f\"SKIP {url.split('/docs/')[1]}: key_fact not verbatim in source\")
skipped += 1
continue
row[\"url\"] = url
f.write(json.dumps(row) + \"\\n\")
kept += 1
print(f\"ok {url.split('/docs/')[1]}: {row['question'][:50]} -> {row['key_fact'][:40]}\")
except Exception as e:
print(f\"SKIP {url}: {e}\")
skipped += 1
print(f\"done -> {kept} kept, {skipped} rejected by curation gate\")
EOF
docker compose exec rag-api python -m app.gen_qa
ok cloud_portal/management/billing/how_billing_works/: What is the starting point ... -> How Billing Works
SKIP cloud_portal/management/billing/overview/: key_fact not verbatim in source
ok cloud_portal/platform/inference/api/wiline-edge-cloud-inference-api/: What is the base URL ... -> https://inference.wiline.com/v1
SKIP cloud_portal/platform/inference/api_keys/: key_fact not verbatim in source
...
done -> 15 kept, 22 rejected by curation gate
15 kept, 22 rejected. That rejection rate is the point: more than half of what the generator\nproduced would have graded the RAG against wrong references. A smaller, trustworthy dataset beats\na bigger, poisoned one.
\nThe generator samples at temperature 0.3, so the exact tally shifts slightly between runs ā here's\na later regeneration that kept 16 of 37; the rejection rate stays brutal either way:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/rag-qa-gen-8dfb631c47f8fc4cb9c506fef0f651d2.png\")
\nFigure 7. The gate at work: 16 kept, 21 rejected on this run. Every surviving label is provably\nin the docs ā real answer-facts like audio/mpeg and the API base URL, not menu paths. (The\ndocumented eval runs below use our first curated set: 15 pairs + the regression = 16 tests.)
Rebuild tests.csv with a rubric that gives the judge the question and the verified\nreference, plus explicit PASS/FAIL rules (different wording is fine; contradiction fails):
{ echo 'question,__expected'; jq -r '[.question, (\"llm-rubric:Question asked: \\\"\" + .question + \"\\\". Reference fact from the official docs: \\\"\" + .key_fact + \"\\\". PASS if the answer correctly addresses the question and does not contradict the reference fact ā different wording is fine. FAIL only if the answer is wrong, contradicts the reference, or fails to answer the question.\")] | @csv' ../app/qa.jsonl; } > tests.csv
Results:
ā 12 passed (75.00%)
ā 4 failed (25.00%)
Duration: 2m 47s (concurrency: 4)
75%. Better ā but before celebrating, read the four failures. And here the series' habit pays\noff one more time, because three of them weren't RAG failures either.
\nThe three failures whose grader reason was just \"No output\" all shared two tells:\ngraderError: true, and a completion of exactly 1024 tokens ā Promptfoo's default\nmax_tokens. Our judge is a thinking model: it spent the whole budget reasoning and got cut off\nbefore emitting the verdict JSON, which Promptfoo counts as FAIL. The fourth \"failure\" got a\nverdict ā but the small judge had literally echoed the schema placeholder (reason: \"string\")\ninstead of grading. Both answers were actually correct.
Two config lines fix it ā a bigger judge, and room to think:
\ndefaultTest:
options:
provider:
id: openai:chat:Qwen3.5-122B
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
# thinking models spend tokens on reasoning before the verdict JSON;
# the 1024 default silently truncated the judge and failed 3 tests
max_tokens: 4096
Results:
ā 14 passed (87.50%)
ā 2 failed (12.50%)
And the row that matters most:
\nā What is the pricing model for GPU hourly usage in Edge Cloud? ā [PASS] ... starting at $0.66/GPU-hour. ā
The regression is green ā the Step-4 conflation is pinned and cannot quietly return. (One\ncosmetic note: the 122B judge sometimes keeps its reason text to a terse ... ā the verdicts\nthemselves are correct, which is what the gate reads.)
You can gate a live RAG service in CI: generated + gate-curated QA, a judge for semantics,\ndeterministic regressions for known bugs ā and you know how to debug the judge itself.
One habit this series drills: re-run before you believe. We did ā and got 75%, with\ndifferent rows failing than before. Billing, which had just passed, now failed with a circular\necho; the TTS answer said \"mp3\" one run and \"unspecified\" the next. The score was a lottery\nbecause the service itself was non-deterministic: generate() passed no temperature, so\nevery eval graded a different roll of the dice at the default 1.0.
One line ends the lottery ā add to the create() call in generate():
temperature=0, # determinism: same question -> same answer
Re-run twice. 13/16 ā 81.25% ā both times, with the same three failures. The number went\ndown and that's the win: 87.5% was partly luck (the flaky rows happened to land right that run);\n81.25% is reproducible. A reader who re-runs your eval should get your number.
\nAfter four rounds of fixing the measurement, what's left is signal ā stable across runs, and\neach one is a different classic RAG failure type:
\n\"iated with subnets\\nData Transferā¦\"). Fixed-size character chunking\nsplit the fact across a boundary, and the half with the answer doesn't rank. Fix class: larger\noverlap, bigger k, or structure-aware chunking.A trustworthy eval isn't the goal; it's the map. It just told us two of three failures are the\nsame class ā facts that don't reach the top-4 retrieved chunks. The cheapest fix in that class is\none character, in retrieve():
def retrieve(question: str, k: int = 6): # was 4 ā two misses said \"not enough context\"
Re-run. Then re-run again, because that's the habit now:
\nResults:
ā 14 passed (87.50%)
ā 2 failed (12.50%)
14/16 ā 87.5% ā twice, same two failures. Read what happened with each:
\nk=6 the chunk with the stats block doesn't rank: this miss is\nsemantic, not budgetary ā the boundary-mangled text (\"iated with subnetsā¦\") embeds poorly.\nMore k can't fix what chunking broke; this one genuinely needs structure-aware chunking.The number is back to 87.5% ā the same as the lucky run ā but this time it reproduces, and\nevery point of it has an explanation.
\n![\"Final](\"https://development-wec.wiline.com/docs/assets/images/rag-eval-final-eb3776d3a5abed5af34d6896bbee06d0.png\")
\nFigure 8. The final run: 14/16, the same two failures every time, regression PASS.
39.5% ā 44.7% ā 75% ā 87.5% (lucky) ā 81.25% (honest) ā 87.5% (earned). Until the very\nlast step, the RAG barely changed ā what changed was the measurement: assertions (substrings ā\njudge), labels (curation gate), the judge's own config (truncation), determinism (temperature).\nOnly then did one retrieval parameter, chosen by the eval, actually move quality. That's the\nlast lesson of the series:
\nPOST /ask with\ndeterministic citations.Parts 1ā4 were the toolkit. This is what the toolkit is for.
\n/ask returns nothing after re-indexingāIf you drop and recreate the Chroma collection while the API is running, the service holds a\nstale collection handle and every query throws:
\nchromadb.errors.InvalidCollectionException: Collection 9b486740-ā¦ does not exist.
The running process cached the old collection's UUID. Restart the service\n(docker compose restart rag-api) ā or resolve the collection per-request instead of at startup.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/rag-stale-handle-1639bfb9ef1938577ff551daa191a889.png\")
\nFigure 9. Re-indexing invalidates open handles ā a classic \"works until you rebuild the index\"\nproduction trap.
Failed to send telemetry event ā¦ capture() takes 1 positional argumentāHarmless chromadb/posthog version clash. Set ANONYMIZED_TELEMETRY: \"False\" in the environment\n(already in our compose).
graderError: true)āCheck the grading record's completion tokens: if it's exactly 1024 ā Promptfoo's default\nmax_tokens ā your thinking-model judge spent the whole budget reasoning and was cut off before\nthe verdict JSON. Set max_tokens: 4096 on the judge provider. Related tell: a small judge\nreturning reason: \"string\" is echoing the schema placeholder instead of grading ā use a bigger\njudge.
Reasoning judges are slow (~minutes for a dozen rows) and imperfect ā expect occasional verdict\nflips between runs. Keep deterministic assertions for known bugs (they're free and never flip)\nand treat the judge score as a trend, not a truth. If a judge row hangs entirely, it's usually a\ntransient network hiccup, not your config ā retry before debugging.
\nThe image needs ~1 GB (chromadb + onnxruntime are the heavy layers). Check df -h / first and\ndocker system df for reclaimable space ā but never prune blindly on a shared box: \"reclaimable\"\nlies when other projects' containers are running.
The honest limitations are the roadmap, and each one traces to a failure you watched happen:\nstructure-aware chunking (the subnets miss survived k=6 ā the boundary-mangled chunk embeds\npoorly, so no retrieval budget rescues it), prompt or model work on generation (the billing\necho: force the model to name the page it's citing, or use a stronger generator), table-aware\nextraction (the pricing strip is still fragment soup ā the regression passes, but \"$1.44/mo\"\nstill bleeds near the GPU line; add a positive assert for the correct figure, not just\nnot-icontains for the wrong ones), and judge calibration (grade the judge against a handful\nof human verdicts). Each one moves the pass rate for a reason you can name ā which, after five\nparts, is the whole point: you don't guess whether your AI works. You measure it, watch it, and\nmake it prove itself.
A customer says your support bot promised them a refund policy that doesn't exist. Your feature\nmade two LLM calls ā classify, then reply. Which one invented it? If you can't answer that,\nyour app is a black box ā and this guide fixes exactly that.
\nYou can now prove a model works (part 1),\nmake its output machine-reliable (part 2),\nand generate a real test set to check it against\n(part 3). But all of that runs offline,\nin CI, on inputs you chose. Production doesn't play along.
\nThis guide closes the gap. We'll self-host Langfuse ā the open-source, self-hostable\nalternative to LangSmith ā trace every real call, auto-score live traffic with an LLM judge,\ndrill into the exact step that breaks, and feed failures back so your part-3 dataset gets\nstronger. Offline eval tells you it worked on your test set; this tells you it works in the wild.
\n\nBy the end you can take any WEC feature and: see inside every call, score quality on real\ntraffic automatically, find the exact call that broke and why, and turn production failures into\nnew test cases. That's the difference between \"it passed CI\" and \"it's holding up in production.\"
\nWe'll keep the running example from part 3 ā a support-ticket classifier on the WEC Inference\nAPI ā and put it under observation.
\nYour app calls the WEC API as usual; Langfuse (self-hosted) quietly captures every call as a\ntrace, an LLM judge scores each one, and the failures flow back into your part-3\ndataset ā so production keeps making your evals stronger:
\n\nWe use Langfuse because it's the open-source leader for LLM observability (MIT-licensed,\nself-hostable), its SDK is OpenAI-compatible so it points at the WEC API unchanged, and ā the\nreason it fits this series ā it has datasets, scoring, and LLM-as-judge evaluators built in.\nThat's what lets your part-3 dataset and the \"score production, feed failures back\" loop be\nfirst-class instead of bolted on.
\nThe trade-off: the production stack is heavy (Postgres + ClickHouse). If you don't need the eval\nfeatures and want lighter, reasonable alternatives are Arize Phoenix (OpenTelemetry-native,\nstrong for offline eval), Helicone (a one-line proxy ā but no evals), or OpenLIT\n(single-binary, OTEL-native). For an eval-driven workflow like this one, the extra containers earn\ntheir keep; for plain tracing, one of those may suit you better.
\nLangfuse's self-host stack is six services (web + worker, ClickHouse, Postgres, Redis, object\nstore), so give it a WEC Instance with ~15 GB disk and 4 GB+ RAM ā don't cram it onto a box\nthat's already busy.
\ngit clone https://github.com/langfuse/langfuse.git && cd langfuse
Generate three secrets and drop them in a .env (the ENCRYPTION_KEY must be hex, or the\nweb container won't boot):
cat > .env <<EOF
NEXTAUTH_SECRET=$(openssl rand -base64 32)
SALT=$(openssl rand -base64 32)
ENCRYPTION_KEY=$(openssl rand -hex 32)
NEXTAUTH_URL=http://<your-instance-ip>:3000
EOF
docker compose up -d
docker compose ps
![\"The](\"https://development-wec.wiline.com/docs/assets/images/langfuse-deploy-ps-597bc541cd07b60e848e6c0dfe6246fc.png\")
\nFigure 1. docker compose ps ā all six services (web, worker, ClickHouse, Postgres, Redis, MinIO) up and healthy.
If the box already runs something on port 3000 or 5432, Langfuse won't start (\"address already\nin use\"). The backing databases don't need host ports at all ā edit docker-compose.yml to move\nlangfuse-web to a free host port (e.g. 3001:3000) and drop the Postgres host mapping. Keeping\nyour databases off the host network is good practice anyway. See Troubleshooting.
Open http://<instance-ip>:3000, create an account (first user is the owner), then an\nOrganization ā Project, and under Settings ā API Keys create a key pair\n(pk-lf-ā¦ / sk-lf-ā¦).
Point the Langfuse SDK at your instance and use its OpenAI drop-in ā it auto-traces any\nOpenAI-compatible call, including the WEC Inference API, with zero extra code around each request:
\npython3 -m venv ~/lf-env && source ~/lf-env/bin/activate
pip install langfuse openai
export LANGFUSE_PUBLIC_KEY=\"pk-lf-ā¦\"
export LANGFUSE_SECRET_KEY=\"sk-lf-ā¦\"
export LANGFUSE_HOST=\"http://<instance-ip>:3000\" # use the port you exposed ā 3001 if you remapped in Step 1
export WEC_API_KEY=\"sk-your-wec-key\"
import os
from langfuse.openai import openai # drop-in: auto-traces every call
from langfuse import get_client
client = openai.OpenAI(
base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"],
timeout=30, # ALWAYS set this ā a slow backend hangs forever otherwise
)
def classify(ticket: str) -> str:
resp = client.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\",
name=\"ticket-classify\", # names the trace in Langfuse
messages=[{\"role\": \"user\", \"content\":
\"Classify this support ticket. Return ONLY JSON with keys \"
\"category (billing/technical/account/other), priority (low/medium/high), \"
f\"summary. Ticket: {ticket}\"}],
)
return resp.choices[0].message.content
if __name__ == \"__main__\":
print(classify(\"I was charged twice this month.\"))
get_client().flush() # push traces before the process exits
Run it, then open Tracing ā the call shows up in seconds with input, output, latency, token\ncounts, and (once we add pricing) cost.
\n![\"A](\"https://development-wec.wiline.com/docs/assets/images/langfuse-first-trace-f013e891fddc5d65b3525d5e1ad59fb7.png\")
\nFigure 2. One ticket-classify call in Langfuse: the prompt, the JSON output, latency (~1.2s),\nand token counts ā captured with zero code wrapped around the request.
The OpenAI client has no default timeout. If the model is slow or the backend stalls, your\napp hangs indefinitely ā no error, no trace. timeout=30 turns that into a fast, visible failure.
One call is easy to log; you don't need Langfuse for that. Tracing earns its keep on multi-step\nfeatures, where it shows the whole chain as one tree so you can see which step broke. Let's make\nthe feature realistic ā classify the ticket, then draft a reply from that category (two WEC calls) ā\nand group them into a single trace with the @observe decorator:
from langfuse import observe, get_client
from app import client, classify # reuse the traced WEC client + classifier
def draft_reply(ticket: str, category: str) -> str:
resp = client.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\",
name=\"draft-reply\",
messages=[{\"role\": \"user\", \"content\":
f\"A customer sent this {category} ticket. Write a short, helpful reply.\\nTicket: {ticket}\"}],
)
return resp.choices[0].message.content
@observe() # groups everything below into ONE trace
def handle_ticket(ticket: str) -> str:
category = classify(ticket) # child span 1
reply = draft_reply(ticket, category) # child span 2
return reply
if __name__ == \"__main__\":
print(handle_ticket(\"I was charged twice this month.\"))
get_client().flush()
Open that trace and you get a tree: handle_ticket ā ticket-classify ā draft-reply, each\nchild with its own input, output, latency, and tokens. Now a failure has an address ā you can\npoint at the exact step that broke instead of guessing (we use this in Step 5).
You can now see inside any LLM call in production ā input, output, latency, tokens ā without\nwriting a line of logging code.
To get real signal (not one call), push your part-3 dataset through the classifier so the\ndashboard has traffic to analyze:
\nimport os, json
from langfuse.openai import openai
from langfuse import get_client
from app import classify # reuse the traced function
for line in open(os.path.expanduser(\"~/tickets_dataset.jsonl\")):
if line.strip():
classify(json.loads(line)[\"ticket\"])
get_client().flush()
Now the Home dashboard shows traces over time, a latency distribution (p50/p95), tokens by\nmodel, and a cost total ā the \"watch it in production\" view.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/langfuse-scores-dashboard-e46938c8695864fa98dc8afa0e488b3a.png\")
\nFigure 3. After replaying the part-3 dataset: traces over time, a latency distribution, tokens\nby model, and a cost total ā the \"watch it in production\" view.
Langfuse auto-computes cost for models it knows (OpenAI, Anthropic), but the WEC Inference API\nreturns no dollar cost and Langfuse doesn't know WEC's model prices ā so cost shows $0. This is\nthe same gap we hit in part 1. Fix it once:\nSettings ā Models ā add a model matching Qwen2.5-3B-Instruct with your per-token input/output\nprices. Now every trace and the dashboard show real spend ā the cost-tracking that Promptfoo\ncouldn't give us.
CI eval checks a fixed set. Here we score every real call automatically with an LLM-as-judge.\nThis step looks like pure configuration ā it's also where we hit, and solved, the best bug of the\nseries.
\nCreate the evaluator via Evaluators ā + New evaluator. It's a 3-step wizard:
\n1. Select Evaluator ā pick the Correctness LLM-as-judge template.
\n2. Set up LLM connection ā the judge needs a model to call, and there's none yet, so click\n+ Add LLM Connection. The dialog has more fields than you'd expect ā here's exactly what each\none needs:
\nopenaiwec-judge) ā just the connection's name in Langfusehttps://inference.wiline.com/v1 ā critical; left blank it hits real OpenAIQwen3.5:9B ā the model choice matters a lot\nhere, as you're about to seewec-judge and model Qwen3.5:9B ā Save.![\"The](\"https://development-wec.wiline.com/docs/assets/images/langfuse-llm-connection-e83545064c144db968575c7d85d0c809.png\")
\nFigure 4. The New LLM Connection dialog: adapter openai, the WEC API Base URL, default\nmodels switched off, and a custom model added ā this is what points the judge at WEC instead of\nreal OpenAI.
3. Run Evaluator ā add a filter Name = ticket-classify, toggle Run on live incoming\nobservations so it scores new traffic, and Execute.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/langfuse-evaluator-config-328cb01c6a061467eca223c1d7266d31.png\")
\nFigure 5. The Correctness evaluator ā score name, the ticket-classify filter, and \"run on\nlive incoming observations.\"
We configured everything, sent fresh trafficā¦ and nothing. No score on any trace. Here's how we\ntracked it down ā you may hit the same wall:
\nRequest timed out after 120000ms. Every run, exactly 120s ā so the judge was firing, its\nLLM call just never came back.Qwen2.5-3B-Instruct, rejected tool calls outright\n(instant 400 ā a missing vLLM config on the backend; more on that below). We switched to\nQwen3.5:9B, which answers a direct tool call in ~3s. Still timed out.curl from the host: 200 in half a second. The same judge-style\nrequest from inside the worker container: ~3s. Model, network, container ā all fine.response_format) so the score comes back machine-readable. Replaying\nthat exact call shape exposed the problem: on this backend, judge-style requests to the\nreasoning model are wildly slow and unpredictable ā the identical request ranged from 3 to\n95 seconds across runs (the model \"thinks\" at length, and JSON mode adds overhead). With the\njudge's longer real prompt, runs blew past the 120s default. Nothing was broken ā it was just\nslower than the timeout.The fix is two lines of configuration. Langfuse reads its LLM timeout from\nLANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS (default 120000). Give it room:
services:
langfuse-worker:
environment:
LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS: \"600000\"
langfuse-web:
environment:
LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS: \"600000\"
docker compose up -d langfuse-worker langfuse-web
# verify it landed inside the container:
docker compose exec langfuse-worker env | grep TIMEOUT
Send a fresh trace through app.py, wait a couple of minutes, and:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/langfuse-score-on-trace-99653e359fac7e126dc097103602a4e5.png\")
\nFigure 6. There it is ā Correctness: 1.00 attached to a live ticket-classify trace, with\nthe judge's reasoning a click away. Every new call now gets scored automatically.
You can now auto-grade real production traffic ā and debug an evaluator that silently produces\nnothing: check its Logs, isolate model vs. network vs. container, match the timeout to reality.
The judge needs tool calling. If your model returns a 400 on any tools request, that's\nusually the serving config, not the model ā on vLLM it's the --enable-auto-tool-choice and\n--tool-call-parser flags. That was our case: we reported it, the platform team enabled the\nflags on Qwen2.5-3B-Instruct the same day, and the 400 disappeared. Only the judge's connection\nis affected ā your app keeps its model.
Scoring is slow (~2 min/score) and that's the backend, not you. The reasoning model \"thinks\"\nat length on judge prompts ā worth reporting to your platform team (reasoning budget caps, or a\nguided-decoding backend like xgrammar on vLLM). The raised timeout keeps scoring reliable meanwhile.
Once tool calling was enabled on Qwen2.5-3B, we tried it as the judge ā scores came back in\n2ā10 seconds instead of ~2 minutes. But they were garbage: the same kind of correct answer got\n1, 1, 0, and 0.05 across four runs, with near-identical rationales attached to opposite\nscores. The small model fills the score format perfectly; it just can't actually evaluate.\nMirror image of part 3's finding: the small\nmodel wins at classifying and loses at judging ā judging is a reasoning task. Keep the judge\non Qwen3.5:9B: slow but coherent.
This is where the span tree pays off. Open a handle_ticket trace and you don't get one opaque\nblob ā you get the whole chain as a tree, each step with its own input, output, latency, and\ntokens:
![\"Drilling](\"https://development-wec.wiline.com/docs/assets/images/langfuse-failing-trace-8fcb77412fc10f743dad85e747382695.png\")
\nFigure 7. The handle_ticket tree ā handle_ticket ā ticket-classify ā draft-reply ā with\nper-step latency and tokens. Every step has an address you can inspect.
That structure is what makes production failures debuggable. Say a customer gets a nonsense reply.\nWithout tracing you'd only know \"the reply was bad.\" With the tree you expand it and read each step\nā for example:
\nticket-classify ā category: \"billing\" ā
correct ā so the classifier isn't the problem.draft-reply ā invented a refund policy that doesn't exist ā ā the bug is in step 2.You've localized the failure to the exact step, input, and output ā you'd fix the draft-reply\nprompt, not waste time on the classifier. Offline eval tells you the pass rate dropped; the trace\ntells you which call, on which input, produced which wrong output, at which step. And now that\nStep 4 scores every call, you don't even have to hunt: filter Tracing by low score and the\nsuspicious traces surface themselves ā often a phrasing your part-3 dataset never covered, which is\nexactly what you capture back next.
You can now pin a production failure to the exact step, input, and output ā no more guessing\nwhich call invented that refund policy.
A production failure is a missing test case ā and Langfuse makes capturing it one click. Create\nthe dataset once (Datasets ā + New dataset, e.g. production-failures), then on any trace click\n+ Add to datasets and pick it. The trace's input and output are prefilled as a new item:
![\"Adding](\"https://development-wec.wiline.com/docs/assets/images/langfuse-dataset-loop-9e09a4627451a945e4df5245e6018fe7.png\")
\nFigure 8. \"Add to datasets\" turns a production trace into a new test case ā input and expected\noutput prefilled ā feeding it straight back into your part-3 dataset.
To automate it, pull the traces you've bookmarked or added to a dataset via the Langfuse SDK, write\n{ticket, category, priority} rows into tickets_dataset.jsonl, and re-run the part-1/part-3\nPromptfoo eval ā now it covers the real-world case.
Now your CI eval is stronger because production found a hole in it. That's the flywheel:\noffline eval ā ship ā observe & score real traffic ā capture failures ā offline eval gets\nstronger.
\nYou've closed the loop: production failures now make your evals stronger ā the flywheel that\nkeeps an AI feature good after it ships.
That's the \"keep it good after shipping\" rung. Offline eval (parts 1ā3) proves it works before\nlaunch; this proves ā and keeps ā it working after.
\ndocker compose upāAnother service holds port 3000 (web) or 5432 (Postgres). The databases don't need host ports ā\nin docker-compose.yml, remap langfuse-web to a free port (3001:3000) and remove the Postgres\nhost mapping. Compose merges ports lists, so an override file with ports: [] won't remove\nthem; edit the compose directly.
Almost always ENCRYPTION_KEY ā it must be openssl rand -hex 32 (64 hex chars), not base64.\nCheck docker compose logs langfuse-web.
$0āLangfuse has no price for WEC's models by default. Add the model + per-token prices under\nSettings ā Models (Step 3).
\nCheck Evaluators ā your evaluator ā Logs first ā an error there tells you it's running and\nfailing, not idle. Then match the error:
\ntool_choice ā tool calling isn't enabled for that model on\nthe serving backend (on vLLM: the --enable-auto-tool-choice / --tool-call-parser flags).\nTwo ways out: point the judge's LLM Connection at a model where it works (Qwen3.5:9B), or ask\nyour platform team to enable the flags ā that's what fixed Qwen2.5-3B-Instruct for us. Either\nway, only the judge changes ā your app keeps its model.Request timed out after 120000ms ā the judge's structured-output call is slower than\nLangfuse's default timeout. Raise LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS (Step 4) and\nconfirm with docker compose exec langfuse-worker env | grep TIMEOUT. If plain curl is fast\nbut scoring still times out, don't chase the network ā the structured-output path is what's\nslow, and that's a backend fix.That's not a hang ā it's the reasoning model generating long hidden chain-of-thought on judge\nprompts (highly variable: the same request can take 3s or 95s), plus structured-output overhead.\nRaising the timeout makes scoring reliable but not fast ā the real fix is platform-side\n(reasoning budget caps, guided-decoding backend).
\nYou can now build, prove, observe, and improve a single WEC feature. The last rung is the real\napp pattern: RAG ā retrieval + generation ā with everything from parts 1ā4 baked in\n(schema-checked output, an eval dataset, and Langfuse tracing over the whole pipeline). That's\nwhere it all comes together.
\n++Your eval passes 100% ā of the two test cases you typed by hand. Real users won't phrase\nthings the way you did.
\nIn part 1 and\npart 2 we built a real eval harness ā\nassertions, schema validation, a model matrix. But two cases can't tell you if your app works;\nthey can only tell you it didn't crash on two inputs.
\nThis guide fixes that. The skill is producing a dataset you can actually trust: we use the\nWEC Inference API to generate labeled tickets, then validate and curate them ā because\ngenerated labels are not automatically correct, and treating them as gospel just moves the bug.\nThe result is real test data at scale. When you finally run it through the part-2 harness, coverage\nsurfaces the failures ā genuine misclassifications and debatable labels ā that two hand-picked\ncases hide.
\n\nEach step feeds the next, ending in a dataset that plugs straight into the part-2 eval:
\n\nA WEC Inference API key ā see Inference ā API Keys.
\njq, curl, and Node.js 22+ (Promptfoo runs via npx).
Your key exported and sanitized (a pasted key often carries invisible characters):
\nexport WEC_API_KEY='sk-your-key'
export WEC_API_KEY=$(printf '%s' \"$WEC_API_KEY\" | LC_ALL=C tr -cd '[:print:]')
If a call returns a 401 / \"invalid API key\", your key is expired or revoked ā generate a\nfresh one in Inference ā API Keys. See Troubleshooting.
Ask the model for tickets with their answers ā the labels are what make it a dataset, not\njust inputs. We reuse the exact schema from part 2 (category + priority enums):
curl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{
\"model\": \"Qwen2.5-3B-Instruct\",
\"temperature\": 0.7,
\"messages\": [{\"role\":\"user\",\"content\":\"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences.\"}]
}' | jq -r '.choices[0].message.content'
{\"ticket\":\"Our server went down unexpectedly and we're not sure what caused it.\",\"category\":\"technical\",\"priority\":\"high\"}
{\"ticket\":\"We need to upgrade our plan but the system is showing an error message when trying to change plans.\",\"category\":\"account\",\"priority\":\"medium\"}
{\"ticket\":\"We're having trouble accessing our database and can't seem to log in.\",\"category\":\"technical\",\"priority\":\"high\"}
... 5 more lines ...
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/dataset-gen-batch-adc9c63dcd4af4c2991e7581ea5908ec.png\")
\nFigure 1. A single generation: 8 diverse tickets, each already carrying its category and priority label.
Clean JSONL, valid labels, real diversity ā and it came back in seconds. We deliberately picked a\nsmall, non-reasoning model (Qwen2.5-3B-Instruct) for generation. Here's why that matters.
Reasoning models return their chain-of-thought in a separate reasoning_content field ā tokens\nyou pay for but never use. Ask a reasoning model like Qwen3.5:9B for a few tickets and compare the\nhidden reasoning against the actual answer:
curl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"Qwen3.5:9B\",\"temperature\":0.7,\"messages\":[{\"role\":\"user\",\"content\":\"Generate 3 realistic customer support tickets for a cloud hosting company. Return ONLY JSONL, keys: ticket, category, priority. No markdown, no fences.\"}]}' \\
| jq '{model, completion_tokens: .usage.completion_tokens, reasoning_chars: (.choices[0].message.reasoning_content|length), answer_chars: (.choices[0].message.content|length)}'
{
\"model\": \"Qwen3.5:9B\",
\"completion_tokens\": 2000,
\"reasoning_chars\": 7116,
\"answer_chars\": 412
}
![\"A](\"https://development-wec.wiline.com/docs/assets/images/dataset-gen-usage-5ee23a331ab19ce10dfddd509162743d.png\")
\nFigure 1b. For just 3 tickets, Qwen3.5:9B wrote 7,116 characters of hidden reasoning to\nproduce 412 of answer ā you pay for all ~2,000 completion tokens. (Counts vary per call; the\nlopsidedness doesn't.) That's why we generate with the small, non-reasoning Qwen2.5-3B ā and it's\nthe same reason those reasoning models stumble on structured output in Step 5's matrix.
Never treat generated data as ground truth without checking it. Save a batch and validate three\nthings: every line parses, every label is in-enum, and the classes are reasonably balanced.
\ncurl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"Qwen2.5-3B-Instruct\",\"temperature\":0.7,\"messages\":[{\"role\":\"user\",\"content\":\"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences.\"}]}' \\
| jq -r '.choices[0].message.content' > tickets.jsonl
# 1) do all lines parse as JSON?
jq -e . tickets.jsonl > /dev/null && echo \"all lines valid JSON\" || echo \"INVALID JSON present\"
# 2) any out-of-enum labels? (no rows printed = all valid)
jq -c 'select((.category|IN(\"billing\",\"technical\",\"account\",\"other\")|not) or (.priority|IN(\"low\",\"medium\",\"high\")|not))' tickets.jsonl
# 3) class distribution
jq -rs 'group_by(.category)[] | \"\\(.[0].category): \\(length)\"' tickets.jsonl
all lines valid JSON
account: 2
billing: 2
other: 1
technical: 3
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/dataset-validate-9b1201646866c2062c5e9aa3d093cfd2.png\")
\nFigure 2. Validation output ā every line parses, no out-of-enum labels, and a reasonable class spread.
The intuitive [\"billing\",...] | index(.category) fails with \"Cannot index array with\nstring\" ā inside the pipe, . is the array, so .category tries to index it. Use\n.category | IN(\"billing\", ...) instead. See Troubleshooting.
One batch of 8 isn't enough. Loop the call a few times (higher temperature for variety across\nbatches), concatenate, then dedupe by ticket text:
\nfor i in $(seq 1 5); do
curl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"Qwen2.5-3B-Instruct\",\"temperature\":0.9,\"messages\":[{\"role\":\"user\",\"content\":\"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences.\"}]}' \\
| jq -r '.choices[0].message.content // empty'
echo \"batch $i done\" >&2
done > tickets_raw.jsonl
# dedupe by ticket text
jq -sc 'unique_by(.ticket)[]' tickets_raw.jsonl > tickets_dataset.jsonl
echo \"raw: $(wc -l < tickets_raw.jsonl) ā deduped: $(wc -l < tickets_dataset.jsonl)\"
raw: 40 ā deduped: 40
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/dataset-scale-dedupe-74ac55864b3da488aa334816e11ede4a.png\")
\nFigure 3. Scaling up: 5 batches ā 40 rows after exact-duplicate removal.
Zero exact duplicates across 5 batches ā a good sign the model isn't just repeating itself.
\nunique_by(.ticket) only catches identical text. Two differently-worded \"server is down\"\ntickets are semantic near-duplicates and will slip through ā catching those needs\nembedding-based clustering, a more advanced step. For a starter dataset, exact dedup is fine.
Note the // empty in the jq filter. If a call fails (revoked key, model timeout), the\nresponse has no content and .choices[0].message.content is null ā a plain jq -r would then\nwrite the literal word null into your dataset, silently poisoning it. // empty drops those,\nso a failed call adds nothing instead of a junk row. Always check the row count afterward.\nSee Troubleshooting.
Validate the full 40 the same way as Step 2 (jq -e ., the enum check, and both distributions).\nOurs came out balanced: technical 15 Ā· account 10 Ā· billing 10 Ā· other 5; priority high 13 Ā·\nmedium 14 Ā· low 13.
You now have a repeatable pipeline for real test data ā 40 validated, deduped, labeled rows\nwhere you used to have two.
Promptfoo reads test cases from a CSV: each column becomes a variable, and the special\n__expected column holds a per-row assertion. We map ticket ā the prompt var and\n__expected ā icontains:<the correct category>. jq @csv handles the commas and quotes\ninside ticket text:
{ echo 'ticket,__expected'; jq -r '[.ticket, (\"icontains:\" + .category)] | @csv' tickets_dataset.jsonl; } > tests.csv
wc -l tests.csv # 41 = header + 40 rows
The dataset is the deliverable ā everything from here is what it unlocks. First payoff: a model\ncomparison you can actually believe. On two hand-typed cases a ranking is noise; on 40 real,\nlabeled rows it's a measurement. Add each WEC model as a provider and run the whole set:
\ncat > promptfooconfig.yaml <<'EOF'
description: \"Ticket classifier ā model matrix on the synthetic 40-row dataset\"
providers:
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen3.5-122B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
const m = output.match(/\\{[\\s\\S]*?\\}/);
return m ? m[0] : output;
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests: file://tests.csv
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
The result is the opposite of what \"bigger is better\" would predict:
\n| Model | Accuracy | Completion tokens / ticket |
|---|---|---|
Qwen2.5-3B-Instruct (non-reasoning) | 85% (34/40) | ~30 |
Qwen3.5:9B (reasoning) | 47.5% (19/40) | ~965 |
Qwen3.5-122B (reasoning) | 15% (6/40) | ~1,013 |
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/dataset-eval-matrix-9ffe20345f2040803fce117f6803a6ff.png\")
\nFigure 4. Same dataset, three models ā the small non-reasoning model wins by a mile.
The small, non-reasoning model wins decisively ā and the bigger the reasoning model, the worse\nit does. The reasoning models aren't dumber at classifying; they bury the JSON in their\nchain-of-thought. Qwen3.5:9B leaks Thinking: ā¦ into the output; Qwen3.5-122B often returns\nan empty answer entirely ā all ~1,000 tokens spent reasoning, nothing parseable left. They also\nburn ~30Ć more tokens per ticket, which is why the full matrix took 12m 31s, almost all of\nit spent waiting on the two big models. For structured output, thinking out loud is a liability,\nnot an advantage ā a counterintuitive finding you'd never get from a leaderboard.
But notice what actually made that finding possible: the dataset. On two hand-typed cases those\npercentages would be coin-flips; 40 labeled rows are what turn \"which model?\" from a guess into a\nmeasurement. The comparison is the reward for building the data ā not a substitute for it.
\nYou can now pick a model with evidence instead of vibes ā something no leaderboard can do for\nyou, because it doesn't know your task.
(The transform grabs the first {ā¦} block with a non-greedy *? match ā cheap insurance for\nmodels that wrap JSON in prose, exactly like part 2.)
category here ā priority is your next assertionThe __expected column grades only the predicted category against the label. The dataset also\ncarries a priority label we don't score yet. To grade both fields at once, add a javascript\nassertion that parses the JSON output and compares priority too ā the exact same pattern, one more\ncheck. Scoring more of what you generated is the cheapest way to make an eval stricter.
Take the winner ā Qwen2.5-3B at 85% (34/40) ā and pull its 6 failing tickets, comparing each\nto its label. (__expected isn't stored as a variable ā Promptfoo consumes it as the assertion ā\nso read the ground truth from the source dataset.) Because the data is freshly generated, your exact\nfailures will differ, but they consistently split into two recognizable kinds ā and neither is\njust \"the model is dumb\". Representative examples:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/dataset-report-541b1a9921d846d5338c1b1547c49bd5.png\")
\nFigure 5. The web report makes it easy to scan which rows failed and open each one.
| Ticket | Label | Why it's tricky |
|---|---|---|
| \"educational discounts for startupsā¦\" | other | Debatable ā it's about pricing, so billing is defensible |
| \"grant a junior dev IAM access to S3ā¦\" | account | Debatable ā access control vs infrastructure |
| \"feedback on the API documentationā¦\" | other | Debatable ā feedback vs technical |
| \"update the card and add a userā¦\" | account | Multi-intent ā genuinely two categories at once |
The rest are ordinary misclassifications. The real, day-to-day lessons:
\nCommit tickets_dataset.jsonl and tests.csv alongside your eval config, and regenerate/grow\nthe dataset as your product changes.
{ \"error\": { \"message\": \"Authentication Error, invalid API key\", \"code\": \"401\" } }
The key is expired, revoked, or malformed. Generate a fresh one in Inference ā API Keys,\nre-export it, and re-sanitize (tr -cd '[:print:]') in case the paste carried an invisible character.
This comes from [\"billing\", ...] | index(.category) ā inside the pipe, . is the array, so\n.category indexes the array. Use the IN() idiom instead:
jq -c 'select((.category|IN(\"billing\",\"technical\",\"account\",\"other\")|not))' tickets.jsonl
![\"The](\"https://development-wec.wiline.com/docs/assets/images/dataset-ts-jq-index-ec85a2581780363d791db73ff2f03fd0.png\")
\nFigure 6. The index(.field) mistake ā inside the pipe, jq tries to index the array with a string.
nullāA null row means a generation call failed but the loop still wrote its (empty) result. Usual\ncauses: an expired/revoked API key, or the model timing out. Fixes: extract with\n.choices[0].message.content // empty so a failed call writes nothing instead of null;\nre-check wc -l after every generation run; and if calls hang, add a per-request timeout\n(curl -m 60) and consider a lighter, faster model for bulk generation.
__expected is null in the results JSONāThat's expected ā Promptfoo consumes the __expected CSV column as the row's assertion, so\nit isn't kept as a variable. To inspect ground-truth labels for failing rows, read them from your\nsource dataset (tickets_dataset.jsonl) rather than the results file.
You now have a real dataset ā but it runs in CI, offline. The next step is watching your app in\nproduction: self-hosting Langfuse to trace every call, track latency and cost, and run\nevals (using this dataset) against live traffic. That's where offline testing becomes real\nobservability ā and it's the next post in the series.
", "url": "https://development-wec.wiline.com/docs/tutorials/synthetic-eval-datasets-wiline-inference/", "title": "Stop hand-writing test cases: generate an eval dataset with the WEC API", "summary": "Two hand-typed test cases aren't an eval. Use the WEC Inference API to generate a labeled evaluation dataset ā then validate and curate it, because generated labels aren't automatically correct. The result is real test data at scale; feed it to your harness and coverage surfaces the misclassifications and debatable labels two cases would hide. Every command and result is real.", "date_modified": "2026-07-06T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "evals", "promptfoo", "inference", "datasets", "synthetic-data", "observability" ] }, { "id": "https://development-wec.wiline.com/docs/tutorials/structured-output-evals-wiline-inference/", "content_html": "++\"Return only JSON\" is one of the most common instructions in production LLM apps ā and one\nof the least reliable. Models wrap JSON in markdown fences, add a friendly sentence, or (if\nthey're reasoning models) narrate their entire thought process around it. Any of those\nbreaks a strict JSON.parse, and your pipeline falls over.
In this guide we build a Promptfoo eval that makes WEC\nInference API models classify support tickets into schema-validated JSON, then harden\nit against real-world messiness ā and use it to pick a model you can actually trust. This is\npart 2 of the evals series (see part 1\nfor first-run setup). Everything here was run live against https://inference.wiline.com.
Every test runs through the same pipeline. The detail that matters most: the transform\nruns on the model's output before the assertion sees it ā that's what lets us recover\nmessy JSON before validating it.
A WEC Inference API key ā see Inference ā API Keys.
\nThe model names used here (GLM-5.2, gemma4, Qwen2.5-3B-Instruct, qwen3.5:9B) come\nfrom the Models Hub ā swap in whatever you've enabled.
Node.js 22+ and your key exported (Promptfoo runs via npx):
nvm use 24
export WEC_API_KEY='sk-your-key'
If your first run errors with Cannot convert argument to a ByteString, your pasted key has\nan invisible character ā see Troubleshooting at the bottom.
We'll do something closer to a real workload than \"capital of France\": classify a support\nticket into category, priority, and a short summary. The key upgrade over a basic\ncheck is the JSON Schema ā it enforces enums, so a model that invents a category fails.
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā structured output (JSON schema)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
The result ā 50%, and the failure is instructive:
\nticket [zai-org/GLM-5.2]
I was charged twice for my instanceā¦ [FAIL] ```json
{ \"category\": \"billing\", \"priority\": \"high\", ā¦ }
```
My VM won't boot after the latestā¦ [PASS] { \"category\": \"technical\", ā¦ }
ā 1 passed (50%) ā 1 failed (50%)
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/json-step1-schema-fail-dc2b9c12c0ee29b0086499d5c76f2fd6.png\")
\nFigure 1. Same model, same prompt, temperature 0 ā one response is clean, the other is\nwrapped in ```json fences, so is-json can't parse it.
The content is correct; the format isn't. is-json validates both \"is it parseable JSON\"\nand \"does it match the schema\" ā and fenced output fails the first test.
transformāA transform runs on the model's output before assertions. Here we strip the fences.\n(This block uses four backticks because the config itself contains triple backticks.)
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā fence-stripping transform\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
return output.replace(/```json\\n?|\\n?```/g, '').trim();
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
Now both pass ā 2/2. The fences are stripped before is-json sees the output.
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/json-step2-transform-pass-14cb388c6725d0aacffeee1bb320f553.png\")
\nFigure 2. The transform removes the code fences, so the previously-failing ticket parses.
transform needs an explicit returnA multi-line transform is a function body, not an expression ā omit return and you get\nTransform function did not return a value. See Troubleshooting.
Rather than clean up after the model, ask the API to only emit valid JSON via\nresponse_format. No transform this time:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā JSON mode (response_format)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
response_format: { type: json_object }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
2/2 PASS with no transform ā looks like JSON mode works.
\n![\"Step](\"https://development-wec.wiline.com/docs/assets/images/json-step3-jsonmode-6cabcc01d214aaa3b14b1a0842eeb720.png\")
\nFigure 3. With response_format: json_object, GLM-5.2 returns clean JSON, no transform needed.
But before trusting response_format, we need to test it across models ā because a single\ngreen result can be luck, not enforcement.
Run the JSON-mode config against all four chat models. This is the payoff ā a\nWEC-specific reliability comparison you can't get from a generic leaderboard.
\ncat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā JSON-mode reliability across models\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
4/8. GLM-5.2 and Qwen2.5-3B pass cleanly; gemma4 and qwen3.5:9B dump their reasoning\n(\"Thinking Processā¦\") instead of JSON ā despite response_format being set.
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/json-matrix-responseformat-d227b3a1ca8bffbf85f89ca2da11eb87.png\")
\nFigure 4. response_format set on all four ā but the two reasoning-oriented models ignore it.
| Model | Result | Tokens (2 calls) |
|---|---|---|
zai-org/GLM-5.2 | ā ā | 202 |
Qwen2.5-3B-Instruct | ā ā | 243 |
gemma4 | āā (reasoning) | 251 |
qwen3.5:9B | āā (reasoning) | 1,757 |
You can now measure which models actually honor JSON mode ā instead of trusting the flag.
response_format doing anything?āDon't assume the reasoning was caused by JSON mode. Re-run the same matrix without\nresponse_format (using the fence-strip transform instead) and compare:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā matrix without response_format\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
return output.replace(/```json\\n?|\\n?```/g, '').trim();
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
Still 4/8 ā the exact same models fail. So response_format changed nothing: on WEC\ntoday it's effectively a no-op (the backend isn't enforcing JSON mode). The reasoning\nbehavior belongs to the models, not to the flag.
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/json-matrix-transform-77179c366ad47c9eb5b00fe2b5d92381.png\")
\nFigure 5. Identical result without JSON mode ā the flag wasn't doing anything.
Lesson worth its own callout: don't attribute a result to a change until you've tested the\nchange in isolation. One extra run turned a wrong conclusion into the right one.
The reasoning models do emit JSON ā it's just buried in prose. Instead of stripping fences,\nextract the first { ā¦ } block from anywhere in the output:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā JSON extraction transform\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
const m = output.match(/\\{[\\s\\S]*\\}/);
return m ? m[0] : output;
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
7/8 ā a big jump. gemma4 and Qwen2.5 fully recover; qwen3.5:9B recovers one of two. The\none remaining failure is telling: that response emitted JSON and then kept reasoning (with\nmore braces), so the greedy { ā¦ } match over-captured and produced invalid JSON.
![\"Step](\"https://development-wec.wiline.com/docs/assets/images/json-matrix-extraction-3b34ee4a32f6990e447d9590d51d8911.png\")
\nFigure 6. Extracting the JSON block recovers the chatty models ā but it's a heuristic, not\na guarantee (note the one over-capture failure).
The terminal table is fine for a quick read, but the web report is where a matrix like this\ncomes alive:
\nnpx -y promptfoo@latest view
The dashboard at the top gives you the model-reliability picture at a glance ā the Pass\nRate (%) bars are the ones to watch here: three models tall, qwen3.5:9B sitting at 50%.\nBelow, each provider is a column you can expand to the full response (that's how we pulled\nthe over-capture failure in Figure 10 ā the terminal truncates it).
![\"The](\"https://development-wec.wiline.com/docs/assets/images/json-view-report-3739e5abd8419ffc8ece8271ff559851.png\")
\nFigure 7. The report's dashboard ā the Pass Rate bars turn the model matrix into a\none-glance ranking, and any cell expands to the full output.
The report's other charts (the prompt-vs-prompt scatter) shine when you're A/B-testing\nprompts, not models ā a good subject for a later post. For a single-prompt, multi-model run\nlike this one, the Pass Rate bars are the chart that matters.
Promptfoo exits non-zero when tests fail, so a one-liner is a gate (put it in a script or\nMakefile, never your interactive shell):
npx -y promptfoo@latest eval -c promptfooconfig.yaml \\
&& echo \"ā
passed\" \\
|| { echo \"ā eval failed ā blocking\"; exit 1; }
![\"The](\"https://development-wec.wiline.com/docs/assets/images/json-ci-gate-71912b506157c96cfd7daa8e5c81f300.png\")
\nFigure 8. A real failure blocks: 7/8 passed, but qwen3.5:9B's over-capture (from Step 6)\ntrips the gate ā the non-zero exit code is the signal CI uses to block a PR.
On GitLab:
\neval:
image: node:22
script:
- npx -y promptfoo@latest eval -c promptfooconfig.yaml
# WEC_API_KEY set as a masked CI/CD variable ā never committed.
Schema-validated JSON is now a CI gate ā a model or prompt change that breaks parsing\nnever ships.
GLM-5.2 and Qwen2.5-3B reliably produce schema-valid JSON\nand are ~8Ć cheaper than the reasoning models. For structured output on WEC, start there.qwen3.5:9B, and gemma4 on the day we tested) narrate their thinking\nand cost far more ā poor fits for strict JSON.transform is your safety net ā fence-stripping recovers the common case; JSON\nextraction recovers most chatty output ā but extraction is a heuristic that can over-capture.response_format is a no-op on WEC today ā verified by isolating it. Don't rely on it;\nvalidate instead.A multi-line transform is compiled as a function body, so it needs an explicit\nreturn. output.replace(...) on its own returns nothing.
options:
transform: |
return output.replace(/.../g, '').trim(); # note the `return`
![\"The](\"https://development-wec.wiline.com/docs/assets/images/json-ts-transform-return-764896c26bc05d65d37eac7808ee06fb.png\")
\nFigure 9. Without return, Promptfoo errors instead of transforming.
If is-json fails with \"Expected output to be valid JSON\" and the response contains a\n\"Thinking Processā¦\" / \"Final Reviewā¦\" narration, you're using a reasoning model. Sometimes it\nemits only reasoning; sometimes ā as below ā it emits valid JSON and then keeps talking,\nso even the Step 6 extraction over-captures and the parse fails. Options: switch to\nGLM-5.2 / Qwen2.5-3B, or accept that the extraction transform is best-effort, not a guarantee.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/json-ts-reasoning-dump-dbfebf9e8d4f5f89e6b2e14b8e0edf50.png\")
\nFigure 10. The failing case expanded in the report: qwen3.5:9B returned valid JSON and\nthen kept reasoning (\"7. Final Reviewā¦\"), so the extraction over-captured and the assertion\nreports \"Expected output to be valid JSON.\"
Your pasted API key contains an invisible Unicode character. Strip it:
\nexport WEC_API_KEY=$(printf '%s' \"$WEC_API_KEY\" | LC_ALL=C tr -cd '[:print:]')
![\"The](\"https://development-wec.wiline.com/docs/assets/images/json-ts-bytestring-0bb4979ce94f0775f1fe314315b38e01.png\")
\nFigure 11. A stray character in the key trips the HTTP header before any request is sent.
Have a structured-output case that still slips through? That's the next test to add.
", "url": "https://development-wec.wiline.com/docs/tutorials/structured-output-evals-wiline-inference/", "title": "Trustworthy JSON: schema-validate your model's structured output", "summary": "LLMs promise JSON and deliver markdown fences and reasoning. Build a Promptfoo eval that classifies support tickets into schema-validated JSON on the WEC Inference API ā with transforms to recover messy output, a model-reliability matrix, and a CI gate. Every command and result is real.", "date_modified": "2026-07-01T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "evals", "promptfoo", "inference", "json", "structured-output", "observability" ] }, { "id": "https://development-wec.wiline.com/docs/tutorials/eval-models-promptfoo-wiline-inference/", "content_html": "\n+You wouldn't ship code without tests ā but most teams ship LLM features on \"looks good\nto me.\" Evaluations (evals) fix that: you define test cases and pass/fail criteria, then\nmeasure your model objectively ā catching regressions, comparing models, and gating\ndeploys.
\nIn this guide you'll build a real eval harness with Promptfoo\npointed at the WEC Inference API ā the same OpenAI-compatible endpoint you call from\nyour apps. Everything here was run live against https://inference.wiline.com; the outputs\nare real.
A WEC (WiLine Edge Cloud) account and an Inference API key ā see\nInference ā API Keys.
\nNode.js 22+. Promptfoo treats Node 20 as end-of-life; with nvm:
nvm install 24 # 24 LTS preferred
nvm use 24
node -v # v24.x
We'll run Promptfoo with npx ā nothing to install globally.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-api-key-a1aca6bfffa7335651ab59ec1c07d8f4.png\")
\nFigure 1. Create a key under Inference ā API Keys, then copy it.
Put your key in the shell:
\nexport WEC_API_KEY='sk-your-key-here'
If your first eval errors with Cannot convert argument to a ByteString, see Troubleshooting #1 below.
The API is OpenAI-compatible, so the model list is a plain GET /v1/models:
curl -s https://inference.wiline.com/v1/models \\
-H \"Authorization: Bearer $WEC_API_KEY\" | jq -r '.data[].id'
zai-org/GLM-5.2
gemma4
Qwen2.5-3B-Instruct
qwen3.5:9B
whisper-large-v3
whisper-medium
whisper-base
The whisper-* models are speech-to-text; the rest are chat models. We'll evaluate the\nchat models, starting with zai-org/GLM-5.2.
Create a config. apiBaseUrl points Promptfoo at WEC; apiKeyEnvar tells it which env\nvar holds your key:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-first-eval-42c0a9a515e6a1e5af7a6cf5c6a30118.png\")
\nFigure 4. Your first passing eval against the WEC Inference API.
icontains is a deterministic assertion ā a case-insensitive substring check. Cheap,\nfast, and perfect for facts with a known answer.
npx -y promptfoo@latest view
This opens a local web UI with every prompt, output, and pass/fail ā far easier to scan\nthan the terminal once you have more than a handful of cases.
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-view-report-b93aa9bcadc4c60e008df1bb5effcd3c.png\")
\nFigure 5. promptfoo view ā the same results in the browser.
Adding a provider is one extra block. Here we add Qwen2.5-3B-Instruct alongside GLM-5.2\nand run the same two test cases against both:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā model comparison\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
- id: openai:chat:Qwen2.5-3B-Instruct
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-two-model-3c6c51ea8839c03dd51d270c78648ec9.png\")
\nFigure 6. Both models pass. GLM-5.2's results are served from cache (0 new requests); only Qwen2.5 makes live calls.
Substring checks can't judge an open-ended answer. A model-graded llm-rubric assertion\nuses a model to score the output against criteria ā and the grader can run on WEC too, so\nyou need no external provider:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā model-graded (LLM rubric)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
defaultTest:
options:
provider: # the grader model
id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
prompts:
- \"Explain what a {{topic}} is in two sentences, for a beginner.\"
tests:
- vars: { topic: VPC }
assert:
- type: llm-rubric
value: \"Describes a VPC as a private, isolated virtual network in the cloud, and is understandable to a beginner.\"
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-llm-rubric-f969ad98a43427eacf9f83f13f2e6dde.png\")
\nFigure 7. The run shows two token lines ā Eval (75) and Grading (350) ā because the rubric check is itself a model call.
Model-graded checks are powerful, but they aren't free.
\nA correct answer that takes 30 seconds is still a production problem. Add a latency\nguardrail to defaultTest (it applies to every case). Always run with --no-cache ā cached\nresponses report ~0 ms and would pass the gate for free:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā latency guardrail\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
defaultTest:
assert:
- type: latency
threshold: 5000 # fail anything slower than 5 s
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
This failed ā the answers were correct (Paris, Tokyo) but both calls took ~33 s,\ntripping the 5 s gate:
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-latency-fail-ab40e8dc4386c6acd496942ee3a490aa.png\")
\nFigure 8. The guardrail fires: correct answers, wrong latency. Duration: 33s.
Bump the threshold to something realistic and re-run:
\nsed -i 's/threshold: 5000/threshold: 60000/' promptfooconfig.yaml
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-latency-pass-f3fa1a1c9cb0a11a87328b2a15ef1f31.png\")
\nFigure 9. With a 60 s threshold the same calls pass easily ā Duration: 2s.
Back-to-back calls to the same model ranged from ~2 s to ~33 s on the same day. Run a\nfew --no-cache passes to get a realistic baseline before locking in your threshold.
The cost assertion doesn't work against WEC (see Troubleshooting #2). Use latency + max_tokens together as your production guardrail instead:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā guardrails (latency + token cap)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
max_tokens: 16
defaultTest:
assert:
- type: latency
threshold: 30000
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-latency-token-cap-968bf5b53dd3edf6d77129f011e4a66f.png\")
\nFigure 11. With a 30 s threshold and a 16-token cap, both cases pass.
Real apps want JSON, not prose. is-json checks the output is valid JSON and matches a\nschema in one assertion:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā JSON schema validation\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- 'Return ONLY a JSON object (no markdown, no prose) with keys \"capital\" (string) and \"population_millions\" (number) for the country {{country}}.'
defaultTest:
assert:
- type: is-json
value:
type: object
required: [capital, population_millions]
properties:
capital: { type: string }
population_millions: { type: number }
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-json-schema-82dcc59881029f9bc0134dcf5a5c5ea2.png\")
\nFigure 12. GLM-5.2 returns compact, schema-valid JSON on the first try.
Here's what you can't get anywhere else ā the same strict-JSON test across all the chat\nmodels on your account:
\ncat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā model matrix\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- 'Return ONLY a JSON object (no markdown, no prose) with keys \"capital\" (string) and \"population_millions\" (number) for the country {{country}}.'
defaultTest:
assert:
- type: is-json
value:
type: object
required: [capital, population_millions]
properties:
capital: { type: string }
population_millions: { type: number }
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
- vars: { country: Brazil }
assert: [{ type: icontains, value: Bras }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-model-matrix-table-fcebe135ddd276ef881fc83a726c6b96.png\")
\nFigure 13a. 12 test cases across 4 models ā 9 pass, 3 fail. qwen3.5:9B emits its chain-of-thought instead of JSON.
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-model-matrix-summary-96e3ce6b439ae8d487aef486120dbe97.png\")
\nFigure 13b. The token breakdown tells the real story: the reasoning model cost 13Ć more for the same task.
| Model | Strict JSON? | Notes | Tokens (3 calls) |
|---|---|---|---|
zai-org/GLM-5.2 | ā all pass | compact JSON | 177 |
gemma4 | ā all pass | integer populations | 207 |
Qwen2.5-3B-Instruct | ā all pass | pretty-printed, precise | 262 |
qwen3.5:9B | ā all fail | reasoning model ā emits its \"thinking\" instead of raw JSON | 2,650 |
Two lessons fall out immediately:
\nqwen3.5:9B is a reasoning model. It \"thinks out loud,\" so it breaks strict JSON\nunless you strip the reasoning or use a response-format constraint. Picking it for\nstructured output would silently fail.That's a model-selection decision you can now defend with data ā for your workloads, not a\ngeneric leaderboard.
\nYou can now compare every model on your endpoint with objective, repeatable numbers ā\naccuracy, latency, and cost side by side.
Promptfoo exits non-zero (100) when any test fails. Verify this first:
npx -y promptfoo@latest eval -c promptfooconfig.yaml > /dev/null 2>&1; echo \"exit code: $?\"
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-exit-code-a3471bde65e7ebf24feaf0d964a3b7eb.png\")
\nFigure 14. Exit code 100 confirms Promptfoo signals failure reliably ā exactly what CI needs.
Wire it up as a gate:
\nnpx -y promptfoo@latest eval -c promptfooconfig.yaml \\
&& echo \"ā
eval passed\" \\
|| { echo \"ā eval failed ā blocking\"; exit 1; }
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-ci-gate-table-7067b62a9b737ed79e3906b6da54ea2a.png\")
\nFigure 15a. The gate re-runs the matrix eval from cache ā same 9/3 result.
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-ci-gate-result-8765593b4f037c480e7fcef844ff6eb4.png\")
\nFigure 15b. Promptfoo exits non-zero, the || branch fires, and the gate blocks.
Never run the exit 1 variant in your interactive shell ā see Troubleshooting #3.
In CI, you don't even need the exit 1 ā the non-zero exit fails the job. On GitLab:
eval:
image: node:22
script:
- npx -y promptfoo@latest eval -c promptfooconfig.yaml
# WEC_API_KEY is set as a masked CI/CD variable ā never commit it.
Now a prompt change that quietly breaks JSON output, or a model swap that doubles latency,\nfails the pipeline before it reaches users.
\nYour eval is now a deploy gate ā regressions fail CI instead of reaching users.
API call error: TypeError: Cannot convert argument to a ByteString because
the character at index 32 has a value of 8232 which is greater than 255.
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-invisible-char-error-540ad4b2ab9242c120ae80870319df01.png\")
\nFigure 2. Every API call errors before reaching the server ā the key never even leaves the machine.
Cause: The pasted key contains an invisible Unicode character ā U+2028 (line separator). Some terminals and clipboard managers insert it silently when copying from certain sources.
\nFix: Strip non-printable characters from the key:
\nexport WEC_API_KEY=$(printf '%s' \"$WEC_API_KEY\" | LC_ALL=C tr -cd '[:print:]')
printf 'len=%s\\n' \"$(printf %s \"$WEC_API_KEY\" | wc -c)\"
# len=25
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-key-strip-1a9c78d8c230c69db3e89eecfdfa24ca.png\")
\nFigure 3. After stripping, the key length confirms no extra characters remain. Re-run the eval ā it passes.
Error: Cost assertion does not support providers that do not return cost
![\"Terminal:](\"https://development-wec.wiline.com/docs/assets/images/promptfoo-cost-error-1615b4a5c8b0973274d32c6d658a2e12.png\")
\nFigure 10. The WEC Inference API doesn't return a cost field in its responses.
Cause: Promptfoo calculates cost by looking up a price-per-token for the model in its built-in price list and multiplying by the tokens used. It has prices for the big hosted models but not for WEC's self-hosted model names, so it has no price to apply and the assertion fails. (Most APIs, including OpenAI, don't return a dollar cost in the response ā tools compute it.)
\nFix: Skip the cost assertion entirely. Track spend from the Total Tokens line Promptfoo prints at the end of every run, and use max_tokens in the provider config to cap per-call token usage.
exit 1 closes your SSH sessionāā eval failed ā blocking
logout
Connection to 10.80.4.212 closed.
Cause: Running || { echo \"ā eval failed ā blocking\"; exit 1; } directly in an interactive shell exits the shell process itself ā which, over SSH, closes the connection.
Fix: Put the gate in a non-interactive context: a Makefile target, an npm script, a git pre-commit hook, or a CI job. In those environments exit 1 fails the job without touching your terminal. If you need to test the gate locally, use || echo \"ā eval failed\" (without exit 1) to see the output safely.
You've gone from \"looks good to me\" to a real, automatable eval harness on WEC ā with\ndeterministic checks, guardrails, schema validation, model-graded rubrics, a model matrix,\nand a CI gate.
\nComing up in this series:
\nllm-rubric, factuality scoring, and\nmulti-judge checks so your grader is trustworthy.+Through Parts 1ā4 you deployed\nOpenClaw, secured it with HTTPS, added Telegram, and made it private over a NetBird\nmesh ā all pointed at OpenAI. This part swaps the model out from under it:\npoint the same agent at WEC Models ā WiLine's own OpenAI-compatible inference ā\nand run it on an open-weight model, Llama 3.1 8B Instruct. Same box, no rebuild\nā just a base-URL, key, and model change via the OpenClaw CLI.
\n\nContinues on the same WEC Instance from Parts 1ā4. WEC Models is an\nOpenAI-compatible endpoint at https://inference.wiline.com/v1; every command\nbelow was run against a live OpenClaw gateway container\n(openclaw-openclaw-gateway-1).
If you self-hosted the agent to begin with, you already care about control ā not\ndepending on someone else's cloud, not being one pricing-page email away from a\nrebuild. But there's a gap in that story: the agent runs on your box, and every\nsingle message it handles is still shipped off to OpenAI to think. Self-hosting\nthe agent and outsourcing the model is only half the job. This part closes it.
\nIt's also just cheaper. WEC's own catalog prices Llama3.1-8B-Instruct at\n$0.15 / 1M input tokens Ā· $0.20 / 1M output tokens. OpenAI doesn't publish a\nstandalone rate for gpt-5.5 (the model OpenClaw defaults to in\nPart 1) any more, but its closest\ncurrent family member ā GPT-5.6 \"Terra,\" pitched by OpenAI as\n\"performance competitive with GPT-5.5\" ā lists at\n$2.50 / 1M in Ā· $15 / 1M out. That's an approximate comparison, not an\napples-to-apples benchmark ā the honest move, as always on this blog, is to\neval it on your own workload\nrather than trust either vendor's number. But the direction is clear enough to be\nworth the ten minutes this takes.
openclaw-openclaw-gateway-1.See Inference ā API Keys if you\ndon't already have one (the same key from\nPart 1 of the evals series\nworks fine here).
\n\nFigure 1. Create a key under Inference ā API Keys, then copy it.
Keep it out of your shell history ā put it in a file instead of exporting it\ndirectly:
\necho 'WEC_API_KEY=sk-wec-...' >> env.local
source <(grep -v '^#' env.local | sed 's/^/export /')
Query the OpenAI-compatible /v1/models endpoint to see what's live on your WEC\nInstance:
curl -s -H \"Authorization: Bearer $WEC_API_KEY\" https://inference.wiline.com/v1/models
The response lists every model available to your key, alongside the id this\ntutorial uses:
\n{\"data\":[
...,
{\"id\":\"Llama3.1-8B-Instruct\",\"object\":\"model\",\"created\":1677610602,\"owned_by\":\"openai\"},
...
]}
Model ids on WEC aren't always the display name shown in the portal ā always take\nthe exact id from this response, not a guess.
Don't hand-edit openclaw.json ā use openclaw config set. It validates the\nwhole file on every write, so a typo fails loudly instead of quietly breaking\nthe gateway. A custom provider needs its base URL, its API style, and its model\nlist declared together, in one write ā the schema rejects a provider that's\nmissing any of the three, so build the full object up front:
docker exec openclaw-openclaw-gateway-1 openclaw config set models.providers.wec '{
\"baseUrl\": \"https://inference.wiline.com/v1\",
\"api\": \"openai-completions\",
\"models\": [
{\"id\": \"Llama3.1-8B-Instruct\", \"name\": \"Llama 3.1 8B Instruct\"}
]
}' --strict-json
Updated models.providers.wec. Change will apply without restarting the gateway.
That registers WEC as a provider, but it's not authenticated yet. Don't put the\nkey straight into that JSON ā OpenClaw's compose setup keeps API keys out of\nenvironment variables entirely and stores them in a dedicated, file-backed\nsecrets directory instead (OPENCLAW_AUTH_PROFILE_SECRET_DIR, mounted into the\ncontainer). Hand the key to OpenClaw's own auth store and let it do the wiring:
echo \"$WEC_API_KEY\" | docker exec -i openclaw-openclaw-gateway-1 openclaw models auth paste-api-key --provider wec
Updated config: $OPENCLAW_HOME/.openclaw/openclaw.json
Backup: $OPENCLAW_HOME/.openclaw/openclaw.json.bak
Auth profile: wec:manual (wec/api_key)
Check that the provider and its auth resolved together, then make the model\nthe default so every agent picks it up without extra config:
\ndocker exec openclaw-openclaw-gateway-1 openclaw models list
docker exec openclaw-openclaw-gateway-1 openclaw models set wec/Llama3.1-8B-Instruct
Model Input Ctx Local Auth Tags
wec/Llama3.1-8B-Instruct text 200k no yes configured
openai/gpt-5.5 text 200k no yes configured,alias:GPT
...
Updated config: $OPENCLAW_HOME/.openclaw/openclaw.json
Backup: $OPENCLAW_HOME/.openclaw/openclaw.json.bak
Default model: wec/Llama3.1-8B-Instruct
![\"Terminal](\"https://development-wec.wiline.com/docs/assets/images/openclaw-wec-models-list-bb96e6c9a1dfe7d0c98be4a6b5b40ae6.png\")
\nFigure 2. wec/Llama3.1-8B-Instruct, authenticated and set as the default model.
OpenClaw can run more than one agent per gateway, so every message needs a\ntarget ā pass --agent, matching whatever openclaw agents list calls yours\n(main, unless you renamed it):
docker exec openclaw-openclaw-gateway-1 openclaw agents list
Agents:
- main (default)
Workspace: $OPENCLAW_HOME/.openclaw/workspace
Model: wec/Llama3.1-8B-Instruct
Routing rules: 0
Send it a message and see who answers:
\ndocker exec openclaw-openclaw-gateway-1 openclaw agent --agent main --message \"What is the capital of France?\"
Paris is the capital of France.
That's WEC, on Llama 3.1, with no OpenAI in the loop.
\n![\"Terminal](\"https://development-wec.wiline.com/docs/assets/images/openclaw-wec-agent-verified-e5125e1d417f258ba73f9d11b78244ff.png\")
\nFigure 3. The agent, running on Llama 3.1 8B Instruct via WEC.
Want it snappier? Turn off thinking mode ā it's a slash command inside the\nchat, not a shell command:
\ndocker exec openclaw-openclaw-gateway-1 openclaw agent --agent main --message \"/think off\"
Thinking mode has been turned off.
Or make it stick for every session:
\ndocker exec openclaw-openclaw-gateway-1 openclaw config set agents.defaults.thinkingDefault \"off\"
Updated agents.defaults.thinkingDefault. No gateway restart needed.
\"custom model providers must declare models\" ā declare baseUrl, api,\nand models together, in one config set call. A custom provider missing any\nof the three gets rejected; you can't add models in a follow-up write.
\"models.providers.wec.apiKey is unresolved in the active runtime snapshot\"\nā don't set apiKey on the provider object; let models auth paste-api-key\nown it (Step 3). Already set one? Remove it:
docker exec openclaw-openclaw-gateway-1 openclaw config unset models.providers.wec.apiKey
\"No target session selected\" ā always pass --agent <id> with\nopenclaw agent. Check the id with openclaw agents list first if you're not\nsure (main, unless you renamed it).
-bash: /think: No such file or directory ā /think is a slash command\nfor inside the chat, not a shell command. Pass it through --message instead,\nas in Step 5.
With the agent running on our own model, the next step is measuring it ā\nevaluating answer quality with Promptfoo + WEC Models.
\n+![\"SQLite\"](\"https://development-wec.wiline.com/docs/assets/images/SQLite370-e02c30e84f9e098787bb57701b0b7232.png\")
Hermes Agent is Nous Research's open-source\n(MIT) AI agent ā \"the agent that grows with you.\" Its standout feature is\npersistent memory: it learns your projects and doesn't forget across\nrestarts. This guide deploys it on the same WEC Instance you already use for\nOpenClaw, points it at a model, and proves the memory survives a full reboot.
\n\nRuns on the same WEC Instance from the OpenClaw series ā no new VM. The installer\npulls its own Python (via uv), Node.js browser tools, and a Playwright Chromium\nautomatically. State lives in ~/.hermes/. The messaging gateway runs as a systemd\nuser service, so it coexists with the OpenClaw Docker stack. Captured live on Ubuntu\n22.04 with hermes v0.17.0.
An agent that forgets on every restart is a toy. Hermes keeps a durable store on\ndisk under ~/.hermes/:
~/.hermes/memories/MEMORY.md ā a human-readable file of facts the agent has\nchosen to remember.~/.hermes/state.db ā a SQLite database holding every session and message,\nwith FTS5 full-text search indexes layered on top so the agent can search its own\nhistory.A deploy, crash, or reboot doesn't wipe what it learned. That's the difference between\na chat demo and an agent you can rely on day to day ā and it's exactly what we test at\nthe end.
\nEverything runs on one WEC Instance. Hermes talks to a model over the WEC Inference API,\nkeeps its memory on local disk, and runs as a systemd user service ā so it restarts itself\nafter a reboot and its memory is still there:
\n\nglm-5.2 works too).curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
![\"Running](\"https://development-wec.wiline.com/docs/assets/images/hermes-install-start-1f3d29fcdd9d66dd1e5c1bc28b3c9e5d.png\")
The installer resolves a Python virtual environment with uv, installs the Node.js\nbrowser tools, downloads a Playwright Chromium, syncs the bundled skills, and drops a\nhermes launcher into ~/.local/bin:
ā Main package installed (hash-verified via uv.lock)
ā All dependencies installed
ā Browser engine setup complete
ā Installed hermes launcher ā ~/.local/bin/hermes
ā Configuration directory ready: ~/.hermes/
ā Starting setup wizard...
It then launches the setup wizard automatically. Reload your shell afterward so the\nhermes command is on your PATH:
source ~/.bashrc
The wizard walks through setup type, terminal backend, and messaging. On a box that\nalready ran OpenClaw, it offers to import the old config first ā you can preview\nwhat would be migrated before anything changes.
\nThe first choice is how to set up Hermes:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/hermes-setup-choice-0db3a3f64faec3d72e3477b075af1901.png\")
We chose Quick Setup first. The Portal login showed a device code and sat at\n\"Waiting for approval (polling)\" ā it needs an active Portal subscription, which this\naccount didn't have. We pressed Ctrl+C to cancel and instead brought our own\nmodel in Step 3 (below) with an OpenAI key. If you do have a Portal plan, complete\nthe login here and you can skip Step 3.
For the rest of the wizard:
\nhermes setup gateway).The wizard applies sensible agent defaults (max iterations, a 50% context-compression\nthreshold) and finishes with a tool-availability summary.
\nOut of the box the default model routes through the Nous Portal / OpenRouter and needs\na key. To use your own provider instead, run:
\nhermes setup model
Then select OpenAI āø OpenAI API, paste your API key when prompted (it's written to\n~/.hermes/.env ā never echoed), accept the default base URL\nhttps://api.openai.com/v1, and pick a model ā we used gpt-5.5:
API key saved.
Base URL [https://api.openai.com/v1]:
Select default model:
ā (ā) gpt-5.5
![\"Choosing](\"https://development-wec.wiline.com/docs/assets/images/hermes-model-select-2b0c5452d2e217ff8692c022badd9b28.png\")
To run inference on WiLine's own endpoint, choose Custom endpoint, set the base URL\nto https://inference.wiline.com/v1, paste your WEC API key, and use a model like\nglm-5.2. Same OpenAI-compatible flow.
Confirm the model and key are healthy:
\nhermes doctor
ā Configuration Files
ā ~/.hermes/.env file exists
ā API key or custom endpoint configured
...
ā Memory Provider
ā Built-in memory active (no external provider configured ā this is fine)
The remaining ā lines in doctor are optional tool integrations (web search,\nDiscord, image gen) ā not needed for this guide.
Start an interactive session:
\nhermes
![\"The](\"https://development-wec.wiline.com/docs/assets/images/hermes-tui-welcome-842b22f5c733599e17b84cde75db4933.png\")
Send it a fact and ask it to remember ā watch for the š§ memory tool to fire:
ā Please save to your long-term memory: my WEC project codename is
Bluefin-7, running on the OpenClaw box.
ā š§ memory ? 0.0s
āā ā Hermes āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā®
Saved to long-term memory: your WEC project codename is Bluefin-7,
running on the OpenClaw box.
ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
![\"Hermes](\"https://development-wec.wiline.com/docs/assets/images/hermes-memory-save-458cefa8aaf46e0d1cc554418a3acc38.png\")
That write creates ~/.hermes/memories/MEMORY.md. Exit with /exit or Ctrl+D.
The TUI submits on newline, so pasting multi-line text can send a half-typed message.\nIf a prompt only catches a fragment, type it on one line by hand.
This is the whole point. The Hermes gateway is installed as a systemd user service\nwith linger enabled, so it comes back automatically after a reboot ā no login\nrequired. Restart the service, then reboot the whole box to make the test airtight:
\nsystemctl --user restart hermes-gateway # restart just the gateway service
sudo reboot # then reboot the entire box
![\"Restarting](\"https://development-wec.wiline.com/docs/assets/images/hermes-reboot-ae228254579427ae3018547803f83c62.png\")
Reconnect once the box is up and confirm the service restarted on its own:
\nsystemctl --user status hermes-gateway --no-pager
ā hermes-gateway.service - Hermes Agent Gateway - Messaging Platform Integration
Loaded: loaded (...; enabled; vendor preset: enabled)
Active: active (running) since Thu 2026-06-25 21:21:54 UTC; 1min 36s ago
![\"systemctl](\"https://development-wec.wiline.com/docs/assets/images/hermes-gateway-status-4da5472c10ec41fe81cfc64298589f10.png\")
Now open a fresh session and ask ā no prior context, brand-new session ID:
\nhermes
ā What's my WEC project codename, and which box does it run on?
āā ā Hermes āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā®
Your WEC project codename is Bluefin-7, and it runs on the
OpenClaw box.
ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
![\"A](\"https://development-wec.wiline.com/docs/assets/images/hermes-reboot-recall-eac2b093a48d12569fa730340f5751d3.png\")
It recalled the fact from disk after a full reboot. Inspect where that durable state\nlives:
\ncat ~/.hermes/memories/MEMORY.md
file ~/.hermes/state.db && du -h ~/.hermes/state.db
User's WEC project codename is Bluefin-7, running on the OpenClaw box.
/home/ubuntu/.hermes/state.db: SQLite 3.x database, ...
176K\t/home/ubuntu/.hermes/state.db
A plain-text MEMORY.md you can read and edit, backed by a SQLite state.db for\nsessions. That's the persistent memory ā and it just outlived a reboot.
When you ask Hermes to remember something, it writes the fact to\n~/.hermes/memories/MEMORY.md ā plain Markdown you can open, read, and even edit by\nhand. Separately, your full conversation history is kept in\n~/.hermes/state.db, a SQLite database. Both live on disk, which is why a\nreboot doesn't lose them.
The simplest way to see what the agent knows is just to read the Markdown:
cat ~/.hermes/memories/MEMORY.md
To look inside the database, note that SQLite's engine is built into Python, so Hermes\nuses it without any extra install. The optional sqlite3 command-line tool ā for\nbrowsing the DB by hand ā isn't installed by default. If you want it:
sudo apt install -y sqlite3
sqlite3 ~/.hermes/state.db '.tables'
![\"Output](\"https://development-wec.wiline.com/docs/assets/images/hermes-sqlite-tables-dbdc5016a3514137534038cac3f6298d.png\")
You'll see sessions and messages tables holding the conversation, plus a family of\nmessages_fts* tables. Those are SQLite FTS5 full-text indexes ā including a\ntrigram variant for fuzzy matching ā that let Hermes search its own past conversations:
CREATE VIRTUAL TABLE messages_fts USING fts5(content);
CREATE VIRTUAL TABLE messages_fts_trigram USING fts5(content, tokenize='trigram');
And each session is a row you can inspect ā here are the two from this guide, the\nsecond one created after the reboot:
sqlite3 -header -column ~/.hermes/state.db \\
\"SELECT substr(id,1,15) AS id, model, message_count FROM sessions ORDER BY started_at;\"
id model message_count
--------------- ------- -------------
20260625_211227 gpt-5.5 8
20260625_212333 gpt-5.5 2
You never need the CLI for Hermes to work ā it's only for inspecting the database\nyourself. Day to day, the human-readable MEMORY.md is all you have to look at.
You now run an agent whose memory survives reboots ā and you know exactly where that\nmemory lives on disk.
The messages table holds the actual text of every turn. Each row has a role ā\nuser (what you typed), assistant (Hermes's reply), or tool (a tool's result) ā so\nyou can replay the whole exchange:
sqlite3 -header -column ~/.hermes/state.db \\
\"SELECT substr(session_id,10,6) AS sess, role, substr(content,1,55) AS content
FROM messages ORDER BY timestamp;\"
sess role content
------ --------- -------------------------------------------------------
211227 user hello, can you hear me?
211227 assistant Yes ā I can hear you. How can I help?
211227 user Please save to your long-term memory: my WEC project...
211227 tool {\"success\": true, \"done\": true, \"target\": \"memory\", ...
211227 assistant Saved to long-term memory: your WEC project codename...
212333 user What's my WEC project codename, and which box does...
212333 assistant Your WEC project codename is Bluefin-7, and it runs...
![\"The](\"https://development-wec.wiline.com/docs/assets/images/hermes-sqlite-messages-12ab80eeb7db27fbf02d5c73b075d553.png\")
The interesting part is how the memory got written. The assistant didn't edit a file\ndirectly ā it emitted a memory tool call, and that tool is what wrote\nMEMORY.md. You can see the exact call recorded in the history:
sqlite3 ~/.hermes/state.db \\
\"SELECT tool_calls FROM messages WHERE tool_calls LIKE '%memory%' LIMIT 1;\"
{\"function\": {\"name\": \"memory\", \"arguments\":
\"{\\\"target\\\":\\\"memory\\\",\\\"operations\\\":[
{\\\"action\\\":\\\"add\\\",
\\\"content\\\":\\\"User's WEC project codename is Bluefin-7, running on the OpenClaw box.\\\"}]}\"}}
![\"The](\"https://development-wec.wiline.com/docs/assets/images/hermes-sqlite-memory-toolcall-53bc3762b14f9ca8e30715118341b792.png\")
So the full chain is: you type ā the agent decides it's worth keeping ā it calls the\nmemory tool with action: add ā that writes the fact into MEMORY.md, while the\nwhole conversation stays in state.db, full-text searchable via FTS5. Two stores, one\ndurable memory.
A memory that survives reboots only matters because of what the agent does with it.\nWith Hermes running, the same install ships these capabilities (you saw them in the\nhermes doctor tool list) ā here's what they unlock day to day:
terminal and code_execution tools let it act,\nnot just chat ā e.g. \"clone this repo, install deps, and run the tests,\" or \"check\nwhy the disk is filling up.\" It does it on the WEC Instance and reports back.cronjob tool runs recurring jobs: a morning\nbriefing, a weekly disk-usage report, or watch a service and alert you when\nit fails ā delivered to your messaging channel. (Part 3.)hermes setup tools.)The rest of this series turns each of these into a hands-on guide (see What's next).
\nQuick Setup (Nous Portal) hangs at \"Waiting for approval (polling)\". The Portal\nlogin needs an active subscription/credit. If you don't have one, Ctrl+C and use\nFull setup with your own key instead (Step 3).
doctor shows \"No API key found in ~/.hermes/.env\". The default model points at\nthe Portal/OpenRouter. Run hermes setup model, pick your provider, and paste the\nkey ā doctor then reports \"API key or custom endpoint configured.\"
Deprecated .env warning on gateway start (e.g. MESSAGING_CWD). Harmless;\nfollow the hint to move the setting into config.yaml when convenient.
Repeating WARNING [Telegram] ā¦ in the gateway logs. If you followed the earlier\nOpenClaw parts and added a Telegram channel, Hermes's setup detected OpenClaw and\nimported those messaging settings into ~/.hermes/.env. Because we skipped\nfinishing messaging here, the gateway has a token but no complete config, so it keeps\nretrying and logging this warning. (A fresh box with no OpenClaw Telegram setup won't\nsee it.) Remove the stray entries and restart:
sed -i '/^TELEGRAM_ALLOWED_USERS=/d; /^TELEGRAM_BOT_TOKEN=/d' ~/.hermes/.env
systemctl --user restart hermes-gateway
If that token was ever exposed, revoke it in Telegram's @BotFather and issue a new\none.
\nNow that Hermes is running with durable memory, the rest of the series turns each\ncapability above into a hands-on guide:
\nhermes setup gateway, allowlists, verify the bot\nresponds), so you can hand it tasks from your phone.Links will appear here as each part ships.
", "url": "https://development-wec.wiline.com/docs/tutorials/self-host-hermes-agent/", "title": "Self-host the Hermes Agent with persistent memory", "summary": "Deploy Nous Research's open-source Hermes Agent on your WEC Instance ā with SQLite-backed persistent memory that survives a full reboot. Real install, model config, and a memory-survives-restart test.", "date_modified": "2026-06-25T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "hermes", "agents", "memory" ] }, { "id": "https://development-wec.wiline.com/docs/tutorials/netbird-mesh-vpn-openclaw/", "content_html": "+![\"NetBird\"](\"https://development-wec.wiline.com/docs/assets/images/netbird-dark-0d2ba9c783792904def84b7e687b8f95.png\")
In Article 2 we put Caddy in front of\nOpenClaw for HTTPS, and in Article 3\nwe added a Telegram channel. The gateway works ā but Caddy is still listening on\n0.0.0.0, reachable by anything that can route to the box. This is the capstone\nof the series: we join the server and your laptop to a NetBird mesh,\nrepoint openclaw.local at the mesh IP, and close the public ports. The same\nhttps://openclaw.local/chat URL keeps working ā but only for your devices.\nEvery command and error below is from the actual run.
Continues from Articles 1ā3 on the same WEC Instance ā Ubuntu, OpenClaw behind\nCaddy 2. We use managed NetBird (the free tier at\napp.netbird.io) with agent 0.73.2 on both the\nserver (Linux, kernel WireGuard) and the client (macOS, userspace). NetBird is\nWireGuard-based and open source; you can also\nself-host the control plane, which is a bigger topic for another day.
NetBird has two parts, and the distinction is the whole point:
\nSo you're building a mesh where Peer A (your laptop) reaches Peer B (the OpenClaw box) by its private 100.x address ā and openclaw.local points there, with Caddy bound to that interface only. The public/LAN side has nothing listening.
The key idea for later: the mesh isn't limited to two peers. Authenticate once, and every new service ā Ollama, a VDI, WordPress ā joins as another peer with its own 100.x address, reachable privately with no public ports to open. You set up the auth model once; you add services forever.
The public internet only ever reaches NetBird's control plane ā and that just\nauthenticates peers, it never carries your app traffic. Your services aren't\n\"firewalled off\" so much as invisible: OpenClaw has no public IP, so there's\nsimply no route to it from the internet. The only way in is to be an\nauthenticated peer on the mesh.
\nIn this tutorial we wire up the first two peers ā your laptop and the\nOpenClaw box. The other services make the point: once auth is set up,\nOllama, a VDI, WordPress each just join the same mesh as another 100.x\npeer ā no public ports, no per-service exposure.
Every NetBird mesh has two roles: peers ā your machines, i.e. the OpenClaw box\nfrom Parts 1ā3 and your laptop ā and a control plane that authenticates those\npeers and issues their join keys. You already have the peer (your OpenClaw box);\nnow you need a control plane for it to join. Two ways to get one:
\nWith Option A you'll run two instances: your OpenClaw box (Parts 1ā3,\nunchanged) and a second, dedicated control-plane box (this template). That's the\ncorrect topology ā a control plane should be its own machine, never sharing with a\nworkload. With Option B there's only your one OpenClaw box. Either way you\nnever rebuild the OpenClaw box ā in Enroll your machines\nyou just install the NetBird agent on it and it joins the mesh as a peer.
The steps below set up Option A. WEC runs the install; you give it an email and a\npublic IP, and a few minutes later you have your own NetBird dashboard on\nhttps://<your-ip>.apps.wiline.cloud with a valid certificate.
The dashboard needs a public address. Under Networks ā Elastic IPs, allocate a\nnew IP or pick a free one, and note its value (ours: 108.60.112.165). It must be\nin the VPC and zone you'll deploy into. Full steps:\nConfigure Elastic IPs.
![\"An](\"https://development-wec.wiline.com/docs/assets/images/netbird-eip-available-b4ec56b4c4946226b7c9e199bd540f61.png\")
This new instance becomes your NetBird control plane ā a separate box from your\nOpenClaw instance, which stays untouched. Open Deploy ā Deploy a VM (full walkthrough:\nDeploy a Virtual Machine).\nThe choices that matter here:
\nLocal Subnet in Princeton-VPC), so the IP can attach afterward.![\"Template](\"https://development-wec.wiline.com/docs/assets/images/netbird-template-config-e03939727bf27b7f6ce4ceb4fc5acdf1.png\")
Finish the wizard and deploy.
\nThe IP isn't bound at deploy time ā you attach it once the VM is up. Go to\nNetworks ā Elastic IPs ā your IP ā āÆ ā Allocate, and select your new instance\n(netbird-tutorial). Only VMs in the same VPC appear in the list.
![\"Allocating](\"https://development-wec.wiline.com/docs/assets/images/netbird-eip-allocate-620e7b57084f55e9e8ca3323494cda6d.png\")
The instance shows Running in Compute ā Instances within a minute ā but the\ntemplate keeps working for a few minutes after that, pulling and starting the\nNetBird containers and bringing up the dashboard. Don't expect the URL to answer\nimmediately.
\nCheck from your laptop over HTTPS (not ping ā ICMP is firewalled off, so a\nfailed ping doesn't mean it's down):
curl -skS -m 10 -o /dev/null -w \"%{http_code}\\n\" https://<your-ip>.apps.wiline.cloud
000 / connection refused ā still provisioning; wait a minute and retry.301 then 200 ā it's up (HTTP redirects to HTTPS).To watch it directly, SSH in (ssh ubuntu@<your-ip>) and confirm the containers are\nrunning and the ports are bound:
cloud-init status
sudo docker ps
sudo ss -tlnp | grep -E ':(80|443)'
It's ready once netbird-dashboard, netbird-traefik, and netbird-server show\nUp and 80/443 are listening.
Open https://<your-ip>.apps.wiline.cloud ā click through the certificate\nwarning if you get one (same self-signed-cert click-through as\nArticle 2).
This is your own NetBird control plane, running on your WEC Instance. On the\nfirst visit it asks you to create the admin account ā enter a name, email,\nand password, click Create, then sign in with those same credentials.\n(That setup screen appears only once; afterwards the dashboard shows a normal\nSign in page.)
\nYou land on the NetBird dashboard ā your private control plane on your own domain:
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/netbird-dashboard-c98eca329c445df75fee98ca55f230e9.png\")
Click Add Peer and NetBird hands you the enrollment command for your server ā\nnote the --management-url pointing at your own domain:
![\"Add](\"https://development-wec.wiline.com/docs/assets/images/netbird-add-peer-09fd78007bcc0279cae91217c1786221.png\")
curl -fsSL https://pkgs.netbird.io/install.sh | sh
netbird up --management-url https://<your-ip>.apps.wiline.cloud
(That second command opens a browser SSO login. For a headless server, add\n--setup-key <KEY> from your dashboard's Setup Keys page instead ā same as the\nmanaged flow, just with --management-url added.)
This is the dashboard where you create the setup keys that enroll machines ā which\nis exactly what we do next.
\nYou have a control plane (your own template dashboard, or managed NetBird). Now put\nthe two machines on the mesh ā the OpenClaw box from Parts 1ā3 and your\nlaptop ā and then close the box's public ports. These steps are the same no\nmatter which control plane you picked.
\nThe commands below use managed NetBird (app.netbird.io) as the example. If you\ndeployed the WEC template instead, it's identical except: create the setup key\nin your own dashboard (https://<your-ip>.apps.wiline.cloud) and add\n--management-url https://<your-ip>.apps.wiline.cloud to netbird up, so the\nagent points at your server instead of the managed cloud.
Prerequisites: OpenClaw running behind Caddy from\nArticles 1ā3 (reachable at\nhttps://openclaw.local), a control plane from the step above, and shell access to\nthe box plus admin on your laptop.
This is the box from Parts 1ā3 (a WEC Instance) ā we're enrolling it as a peer,\nwhich is separate from any control-plane server. NetBird ships a one-line installer:
\ncurl -fsSL https://pkgs.netbird.io/install.sh | sh
Gotcha (real one): on Ubuntu the install triggers the needrestart dialog ā\na blue \"Which services should be restarted?\" box. Tab between buttons can get\nswallowed by some SSH/terminal setups; use arrow keys to move, Space to\ntoggle, Enter to confirm (<Ok> is the default). The package is already in\nplace by the time this box appears, so even if you Ctrl+C out of it, the agent\nis installed ā re-running the installer just tells you so:
NetBird seems to be installed already, please remove it before proceeding
(If you re-run the installer after this tutorial, you'll instead see\nNetBird service is running, please stop it before proceeding ā the agent's\nalready up, so there's nothing to reinstall.)
Confirm the version:
\nnetbird version
# 0.73.2
A setup key is the token that enrolls a device into your mesh ā ideal for a\nheadless server (no browser SSO needed).
\nA few things will try to send you the wrong way on a fresh account ā here's the\nstraight line:
https://app.netbird.io/setup-keys.https://app.netbird.io/setup-keys.openclaw-tutorial, toggle Reusable (so it works for the laptop too), defaults otherwise.![\"Creating](\"https://development-wec.wiline.com/docs/assets/images/netbird-setup-key-ade9af748bded2998a18890a1ebde93b.png\")
Connect the VM (replace with your key):
\nsudo netbird up --setup-key <YOUR_SETUP_KEY>
Anyone with a reusable key can add devices to your network. Don't paste it into\nchats or commits, and delete it from the dashboard once your devices are\nenrolled (the existing peers stay connected).
Check the connection:
\nnetbird status
![\"netbird](\"https://development-wec.wiline.com/docs/assets/images/netbird-vm-status-8e8088e406d7392203f1eb5da63b4258.png\")
That NetBird IP (100.87.239.229) is the address everything below points\nat. Peers count: 0/0 is expected ā nothing else has joined yet.
Install the NetBird client on whatever you're using, then connect with the\nsame reusable setup key. The connect command (netbird up --setup-key ā¦) is\nidentical on every OS ā only the install differs.
The captured output below is from a Mac. The Windows and Linux tabs give the\nequivalent install; the netbird up, netbird status, and ping steps are the\nsame everywhere (Windows uses ping -n instead of -c).
Install via Homebrew, then start the service and connect:
brew install netbirdio/tap/netbird
sudo netbird service install
sudo netbird service start
netbird up --setup-key <YOUR_SETUP_KEY>
You may see a one-off gRPC warning as the daemon's socket comes up a beat after\nthe client calls it ā it ends in Connected, so it's harmless:
WARNING: ... dial unix /var/run/netbird.sock: connect: no such file or directory
Connected
Download and run the NetBird Windows installer from\napp.netbird.io (Add Peer ā Windows) or\npkgs.netbird.io. It installs a system-tray app and the\nnetbird CLI. Then, in PowerShell as Administrator, connect with your key:
netbird up --setup-key <YOUR_SETUP_KEY>
(You can also click Connect from the tray icon and authenticate via SSO, but\nthe setup key is the same one-shot flow as the other platforms.)
Same one-line installer as the VM, then connect:
curl -fsSL https://pkgs.netbird.io/install.sh | sh
sudo netbird up --setup-key <YOUR_SETUP_KEY>
Verify the two nodes see each other, and that the tunnel carries traffic:
\nnetbird status
ping -c 3 100.87.239.229 # Windows: ping -n 3 100.87.239.229
![\"Mac](\"https://development-wec.wiline.com/docs/assets/images/netbird-mac-status-857d291924930e23af4e6550b1845ec7.png\")
Peers count: 1/1 ā that peer is the VM. In the dashboard, both machines now\nshow up under Peers (because we enrolled them with a setup key rather than an\nSSO login, they appear under Servers rather than User Devices):
![\"The](\"https://development-wec.wiline.com/docs/assets/images/netbird-peers-a54440db77e8715febacc8d410c24da5.png\")
This path is going through a NetBird relay (note Interface type: Userspace\non the Mac). It's fine for a web UI, and NetBird will often upgrade to a direct\npeer-to-peer link when the networks allow. Latency to a relayed peer is not a\nred flag.
openclaw.local at the mesh IPāFrom Article 2, your hosts file maps\nopenclaw.local to the VM's LAN IP (10.80.4.212). Swap it for the mesh IP\n(100.87.239.229). The file path and edit command differ by OS:
sed on macOS needs the empty -i '':
grep openclaw /etc/hosts
# 10.80.4.212 openclaw.local
sudo sed -i '' 's/^10\\.80\\.4\\.212 openclaw\\.local/100.87.239.229 openclaw.local/' /etc/hosts
grep openclaw /etc/hosts
# 100.87.239.229 openclaw.local
GNU sed takes -i with no argument:
grep openclaw /etc/hosts
sudo sed -i 's/^10\\.80\\.4\\.212 openclaw\\.local/100.87.239.229 openclaw.local/' /etc/hosts
grep openclaw /etc/hosts
# 100.87.239.229 openclaw.local
The hosts file is at C:\\Windows\\System32\\drivers\\etc\\hosts and needs admin.\nIn PowerShell as Administrator:
$h = \"$env:WINDIR\\System32\\drivers\\etc\\hosts\"
(Get-Content $h) -replace '^10\\.80\\.4\\.212 openclaw\\.local','100.87.239.229 openclaw.local' | Set-Content $h
Select-String openclaw $h
(Or open that file in Notepad run as Administrator and change the IP by hand.)
Test over the mesh ā before touching any ports:
\ncurl -kI https://openclaw.local/
HTTP/2 200
...
via: 1.1 Caddy
via: 1.1 Caddy over the mesh IP ā the chat now flows through NetBird. Load\nhttps://openclaw.local/chat?session=main in the browser to confirm it behaves\nexactly as before. The TLS cert still matches because the hostname is unchanged;\nonly the IP behind it moved.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/netbird-chat-mesh-409321edd81ea8d59656891246bc0558.png\")
Here's the state we're fixing. On the VM:
\nsudo ss -tlnp | grep -E ':(80|443)'
LISTEN 0 4096 0.0.0.0:443 ... docker-proxy
LISTEN 0 4096 0.0.0.0:80 ... docker-proxy
Caddy listens on 0.0.0.0 ā every interface, including the LAN (and any NAT\nin front of it).
Your first instinct is ufw deny 443. It won't work. Caddy's ports are\npublished by Docker, which writes its own iptables rules that bypass ufw's\nINPUT chain. You'd add ufw rules, feel safe, and 443 would stay wide open. On\nthis box ufw was even inactive ā and enabling it would have changed nothing\nfor the Docker-published ports.
The reliable fix is to bind the published ports to the mesh IP instead of\n0.0.0.0, so Caddy only ever listens on wt0. Edit the Caddy service in\ndocker-compose.override.yml (the override from Article 2):
cd /home/ubuntu/openclaw
cp docker-compose.override.yml docker-compose.override.yml.bak
Change the ports block from:
\n ports:
- \"80:80\"
- \"443:443\"
to bind to the mesh IP:
\n ports:
- \"100.87.239.229:80:80\"
- \"100.87.239.229:443:443\"
Recreate just Caddy and re-check:
\ndocker compose up -d caddy
sudo ss -tlnp | grep -E ':(80|443)'
![\"ss](\"https://development-wec.wiline.com/docs/assets/images/netbird-ss-after-0172e6fc62bc667a141b1f6528526e73.png\")
The ports are gone from 0.0.0.0 ā they live on the mesh interface only.
Because the bind targets the mesh IP, NetBird must be up before Caddy starts\n(e.g. after a reboot, wt0 needs its IP before Docker binds to it). The\nrestart: unless-stopped policy already covers this ā Caddy retries until the\ninterface exists ā but it's worth knowing if you ever see Caddy crash-looping\nright after boot.
Both directions, from the actual run.
\nStill reachable over the mesh (laptop):
\ncurl -kI https://openclaw.local/
# HTTP/2 200 ... via: 1.1 Caddy
No longer reachable on the LAN (VM, hitting its own LAN IP):
\ncurl -kI --connect-timeout 5 --resolve openclaw.local:443:10.80.4.212 https://openclaw.local/
![\"On](\"https://development-wec.wiline.com/docs/assets/images/netbird-refused-0944b25666489470ae7b961e2d6243c8.png\")
That's the result: same URL, reachable only through your mesh, refused\neverywhere else.
\nYou can now take any self-hosted service off the public internet without breaking how you\nuse it.
Delete the reusable setup key now that both devices are enrolled ā\nSetup Keys ā delete in the dashboard. Your peers stay connected; you're just\nclosing the door you used to add them. To remove a device later, delete it under\nPeers ā User Devices and run netbird down on that machine.
needrestart dialog won't take Tab. Use arrow keys / Space / Enter. The package is already installed by the time the box shows.netbird.sock warning on first up. The daemon's socket lags the client by a beat. If it ends in Connected, ignore it.ufw rules don't block the gateway. Docker-published ports bypass ufw ā bind the port to a specific IP (as above) or use Docker's own DOCKER-USER chain.wt0 had the mesh IP. restart: unless-stopped recovers it; or order Docker after netbird.service.Interface type and NetBird's peer detail for the connection type.That closes the Self-hosting OpenClaw series:
\nYou now have a self-hosted agent that's encrypted in transit, paired to your\ndevices, reachable from your phone, and invisible to the public internet ā all on\none small WEC Instance.
\nReady for an agent that remembers? Start the Self-hosting Hermes series ā\nSelf-host the Hermes Agent with persistent memory ā\ndeployed on this same WEC Instance, with context that survives restarts.
", "url": "https://development-wec.wiline.com/docs/tutorials/netbird-mesh-vpn-openclaw/", "title": "Make OpenClaw private with a NetBird mesh VPN", "summary": "Put OpenClaw on a private mesh with NetBird, repoint your hostname at the mesh IP, and close the public ports ā so the same chat URL works only for your devices. Real commands and gotchas from a live run.", "date_modified": "2026-06-24T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "openclaw", "netbird", "vpn", "mesh", "security" ] }, { "id": "https://development-wec.wiline.com/docs/tutorials/add-telegram-channel-openclaw/", "content_html": "+You've deployed OpenClaw and\nsecured it behind HTTPS. Now make it\nusable ā talk to your agent from your phone via Telegram. We create a bot,\nconnect it, clear OpenClaw's pairing gate, and get a real reply. Every command and\ngotcha below is from an actual run.
\n\nContinues from Parts 1ā2 on the same WEC Instance ā OpenClaw 2026.6.8. Telegram\nuses polling (the gateway polls Telegram's API), so no public webhook or domain\nis required ā this works on a private box.
Your phone talks to a Telegram bot; the OpenClaw gateway polls Telegram for\nmessages, runs them through your model, and replies ā no inbound ports needed.
\n\n~/openclawweb.telegram.org. (You can't create a bot from the public t.me web page; see Troubleshooting.)In the Telegram app, tap search and type BotFather, then open the official\naccount ā the one with the blue checkmark.
![\"Searching](\"https://development-wec.wiline.com/docs/assets/images/tg-step1a-search-922bcf2cacaf250ac10c1a03546f240b.png\")
Tap Start ā BotFather replies with the commands it understands (you'll see\noptions like /newbot, /mybots, /token, and more). Send /newbot, then\nfollow its two prompts:
My OpenClaw Agentbot, e.g. wiline_openclaw_botBotFather confirms with \"Done! Congratulations on your new bot\" and gives you the\ntoken ā the line after Use this token to access the HTTP API:, like\n1234567890:AA....
![\"BotFather](\"https://development-wec.wiline.com/docs/assets/images/tg-step1c-token-0bec47ec554ad66500f6d053bd0a4810.png\")
Anyone with it controls your bot. Blur it in the screenshot above, and /revoke\nit in BotFather if it ever leaks.
On the VM, in ~/openclaw, register the channel with your token:
docker compose run --rm openclaw-cli channels add --channel telegram --token \"<your-bot-token>\"
# Added Telegram account \"default\".
Restart the gateway so it picks up the channel, then check status:
\ndocker compose restart openclaw-gateway
docker compose run --rm openclaw-cli channels status
# - Telegram default: enabled, configured, running, mode:polling, token:config
mode:polling means the gateway is actively polling Telegram ā no inbound port or\nwebhook required.
![\"channels](\"https://development-wec.wiline.com/docs/assets/images/tg-step3-status-6393f1690ec9b270312c2f5e3a25f137.png\")
Open your bot (t.me/<your-bot-username>), tap Start, and send a message.
OpenClaw treats unknown senders as untrusted ā so instead of replying, the bot\nreturns a pairing code and your Telegram user id. This is the same security\npattern as device pairing in Part 2.
\n![\"Bot](\"https://development-wec.wiline.com/docs/assets/images/tg-step4-pairing-94e777f1790fbeadab0cd42be384b80b.png\")
On the VM, approve the code from the bot's message:
\ndocker compose run --rm openclaw-cli pairing approve telegram <pairing-code>
# Approved telegram sender <id>.
# Command owner configured telegram:<id> (commands.ownerAllowFrom was empty).
Approving the first user as command owner changes config that requires a\ngateway restart ā OpenClaw does this automatically (you'll see\nconfig change requires gateway restart then the Telegram provider restart in the\nlogs). Wait a few seconds for it to come back before messaging again.
Back in Telegram, just talk to it ā no extra setup needed; OpenClaw with your model\nanswers conversationally out of the box. Try something like:
\nIn one sentence, what's the point of putting a reverse proxy in front of my agent?
It replies with a real answer ā the full chain is live: your phone ā Telegram ā\ngateway ā OpenAI ā reply. Your self-hosted agent is now in your pocket. (Giving it\ntools and skills comes in later guides; conversational chat works immediately.)
\n![\"The](\"https://development-wec.wiline.com/docs/assets/images/tg-step6-working-7acac4effbd1c03d0657534125e3ea3b.png\")
Your agent is now in your pocket ā a private Telegram line to your own AI, no public\nwebhook needed.
Opening t.me/BotFather in a plain web browser shows a Start button that does\nnothing ā that page is just the public bot card and needs a logged-in Telegram\nclient to hand off to. Use the desktop/phone app, or log in at\nweb.telegram.org (link a desktop device by scanning the QR from your phone:\nSettings ā Devices ā Link Desktop Device), then message BotFather there.
Not an error ā it's the pairing gate. Unknown senders must be approved first\n(Step 5). After approval, the bot chats normally.
\nProxy headers detected from untrusted addressāIf you followed Part 2, the gateway logs this because Caddy sits in front of it.\nIt's a warning, not a failure. To silence it and restore correct client\ndetection, set gateway.trustedProxies to your proxy's network.
event loop degraded under load.\nFor a busy bot, give it more CPU/RAM.You've finished Part 3 of the Self-hosting OpenClaw series. Next:
\nThen start the Self-hosting Hermes series ā Self-host the Hermes Agent with persistent memory ā for an agent whose context survives restarts.
\nCompanion files for the series live in the WiLine manifests repo (link coming with the repo).
\nRemove the Telegram channel (keeps the rest of OpenClaw running):
\ndocker compose run --rm openclaw-cli channels remove --channel telegram
docker compose restart openclaw-gateway
And /revoke the bot token in BotFather if you're done with it.
+In Article 1 we got OpenClaw running ā\nbut only over plain HTTP, with an allowInsecureAuth workaround. Here we put\nCaddy in front of it as a reverse proxy: real HTTPS,\ndevice-paired auth, and the gateway's raw ports closed so the proxy is the only\nway in. Every command and error below is from the actual deploy.
Continues from Article 1 on the same WEC Instance ā Ubuntu 22.04.5 LTS, OpenClaw\n2026.6.8. We use Caddy 2 (caddy:2). Because this box has no public domain,\nwe use Caddy's internal CA (tls internal) for self-signed HTTPS; the\nGoing to production section shows the one-line swap to real\nLet's Encrypt certificates once you have a domain.
Caddy terminates HTTPS and reverse-proxies to the OpenClaw gateway over Docker's\ninternal network. The gateway stops publishing its own ports ā the only entry\npoint becomes Caddy on :443.
~/openclawopenclaw.local + a hosts entry.cd ~/openclaw
docker compose ps
You should see openclaw-gateway Up (healthy), publishing 18789/18790/3978.\nThat's our starting state: working, but HTTP-only.
Caddy's config: serve HTTPS and proxy everything to the gateway. tls internal\nuses Caddy's own local CA ā no public domain needed.
cat > Caddyfile <<'EOF'
# Caddy selects its certificate by SNI (the hostname in the TLS handshake).
# A bare IP sends no SNI, so we use a hostname. With a public domain, put your
# domain here and drop `tls internal` for automatic Let's Encrypt.
openclaw.local {
tls internal
reverse_proxy openclaw-gateway:18789
}
EOF
The proxy target is openclaw-gateway:18789 ā Caddy reaches the gateway by its\nCompose service name over the shared Docker network.
docker compose auto-merges a docker-compose.override.yml, so we add a caddy\nservice without touching OpenClaw's official compose file:
cat > docker-compose.override.yml <<'EOF'
services:
caddy:
image: caddy:2
restart: unless-stopped
ports:
- \"80:80\"
- \"443:443\"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:
EOF
caddy_data persists the local CA + certs across restarts.
docker compose up -d
docker compose ps
Caddy comes up alongside the gateway with 443 published, and generates its\ninternal CA + a certificate on first run.
![\"Caddy](\"https://development-wec.wiline.com/docs/assets/images/caddy-step4-up-d53a3b63c4e346428a8e733249b2049a.png\")
docker compose up -d also starts the one-shot openclaw-cli helper; it idling\nhere is harmless ā we invoke it with docker compose run --rm when needed.
Hit the gateway's health endpoint through Caddy. --resolve makes curl send\nthe hostname (SNI) while pointing at the box:
curl -k --resolve openclaw.local:443:127.0.0.1 https://openclaw.local/healthz
# {\"ok\":true,\"status\":\"live\"}
tlsv1 alert internal error?That happens if you used a bare IP as the site address ā see\nTroubleshooting #1. Use a hostname.
Now traffic arrives over https://openclaw.local. Two config changes:
allowedOrigins (the Control UI rejects unknown origins).allowInsecureAuth back off ā HTTPS is a secure context, so the\nArticle 1 workaround is no longer needed.docker compose run --rm --no-deps --entrypoint node openclaw-gateway \\
dist/index.js config set --batch-json '[{\"path\":\"gateway.controlUi.allowedOrigins\",\"value\":[\"https://openclaw.local\"]},{\"path\":\"gateway.controlUi.allowInsecureAuth\",\"value\":false}]'
docker compose restart openclaw-gateway
Open https://openclaw.local/ in your browser (on a private box, first map\n10.80.4.212 openclaw.local in your machine's hosts file ā /etc/hosts on\nmacOS/Linux, or C:\\Windows\\System32\\drivers\\etc\\hosts edited as Administrator\non Windows).
You'll hit a certificate warning first ā \"Your connection is not private\"\n(NET::ERR_CERT_AUTHORITY_INVALID). This is expected with tls internal: the\ncertificate is signed by Caddy's local CA, which your operating system doesn't\ntrust. Click Advanced ā Proceed to openclaw.local (unsafe) to continue. (With a\nreal domain + Let's Encrypt, the certificate is publicly trusted and this warning\nnever appears ā see Going to production.)
![\"Browser](\"https://development-wec.wiline.com/docs/assets/images/caddy-step7-cert-warning-f6015d4f4498bdd7f495c01e08e91cf5.png\")
Now the page loads ā but with allowInsecureAuth off, the gateway requires\ndevice pairing before this browser can use the Control UI. That's the secure\nflow doing its job. The screen shows a request id; approve it on the host:
docker compose run --rm openclaw-cli devices approve <request-id-from-the-screen>
![\"Device](\"https://development-wec.wiline.com/docs/assets/images/caddy-step7-pairing-95202988ec2160fdef5dcd2e33fce67f.png\")
Back in the browser, click Connect again ā it pairs and loads the dashboard.
\nThe gateway still publishes 18789 to the host, so http://<vm-ip>:18789 is\nstill reachable in plain HTTP, bypassing Caddy. Close it: stop publishing the\ngateway's ports so Caddy on :443 is the only door. Caddy still reaches it over\nthe internal network.
cat > docker-compose.override.yml <<'EOF'
services:
openclaw-gateway:
# Unpublish the gateway's host ports ā Caddy reaches it internally, so the
# only entry point is HTTPS on :443.
ports: !reset []
caddy:
image: caddy:2
restart: unless-stopped
ports:
- \"80:80\"
- \"443:443\"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:
EOF
docker compose up -d
ports: !reset [] clears the ports the base compose published.
curl -s -m 5 http://<vm-ip>:18789/healthz ; echo \" <- direct HTTP (should FAIL)\"
curl -sk --resolve openclaw.local:443:127.0.0.1 https://openclaw.local/healthz ; echo \" <- via Caddy (should work)\"
The first returns nothing (connection refused) ā the insecure door is gone. The\nsecond still returns {\"ok\":true,\"status\":\"live\"}.
Your OpenClaw is now reachable only over HTTPS, device-paired.
\n![\"OpenClaw](\"https://development-wec.wiline.com/docs/assets/images/caddy-control-ui-https-814c25fc8dd38568d2226856190421ba.png\")
You can now put real HTTPS and auth in front of any self-hosted service ā and prove the\nback door is closed.
tlsv1 alert internal errorāHit when the Caddyfile site address was a bare IP (https://10.80.4.212):
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error
Caddy's logs showed the cert was obtained fine. Cause: TLS picks a certificate\nby SNI (the hostname in the handshake), but connections to a bare IP send no\nSNI ā so Caddy can't match a cert and aborts. Fix: use a hostname site address\n(Step 2). A reader with a real domain never hits this ā the domain is the SNI.
\nOn the first HTTPS connect after turning off allowInsecureAuth:
Device pairing required ā This browser needs one-time approval from the Gateway
host before it can use the Control UI.
Cause: this is the correct secure flow ā with insecure auth off, each new\nbrowser must be approved. Fix: docker compose run --rm openclaw-cli devices approve <request-id>. (The CLI may log scope upgrade pending approval ā¦ using local fallback but still completes the approval.)
NET::ERR_CERT_AUTHORITY_INVALIDāExpected with tls internal ā the cert is signed by Caddy's local CA, which your\nOS doesn't trust. Click through on a private box. With a public domain + Let's\nEncrypt (below), the cert is trusted and there's no warning.
With a real domain, swap two lines in the Caddyfile ā point it at your domain and\ndrop tls internal:
- openclaw.local {
- tls internal
+ openclaw.example.com {
reverse_proxy openclaw-gateway:18789
}
Then point the domain's DNS A record at the VM's public IP and open 80 + 443\nto the internet. Caddy automatically provisions and renews a Let's Encrypt\ncertificate ā trusted, no warning, no --resolve or /etc/hosts needed. Update\nallowedOrigins to https://openclaw.example.com.
:443 is exposed; the gateway has no host\nports (Step 8).openclaw devices).https://openclaw.local.OPENCLAW_GATEWAY_TOKEN if it has ever been\nshown (screenshots, screen-shares).You've finished Part 2 of the Self-hosting OpenClaw series. Next:
\nCompanion files (Caddyfile, docker-compose.override.yml) live in the WiLine\nmanifests repo (link coming with the repo).
To remove just Caddy and reopen the direct ports, delete docker-compose.override.yml\nand Caddyfile, then docker compose up -d. To stop everything:
docker compose down
Your AI assistant doesn't have to live in someone else's cloud.
\nDeploy a self-hosted OpenClaw AI agent on a WEC Instance with\nDocker Compose ā from spinning up the VM to an agent that actually answers, using\nyour own model API key. Every command, version, and error below was captured\nfrom a real deployment on a WEC Instance.
\n\nYou bring your own model API key (OpenAI in this guide); WiLine handles the\nhosting. This was reproduced on a WEC Instance ā Ubuntu 22.04.5 LTS, 8 vCPU,\n15 GiB RAM, 25 GB free disk. OpenClaw version 2026.6.8, image\nghcr.io/openclaw/openclaw:latest.
A single OpenClaw gateway container, configured with your model provider,\nreachable through its web Control UI and CLI.
\n\n~/.ssh/id_ed25519 or similar)Spin up a WiLine VM that meets these requirements:
\n| Requirement | Value |
|---|---|
| OS | Ubuntu 22.04 LTS (or 24.04 LTS) |
| Compute | ā„ 2 vCPU / 4 GB RAM ā OpenClaw needs ā„ 2 GB; the first run can be OOM-killed (exit 137) on 1 GB hosts |
| Disk | 30 GB NVMe ā 10 GB fills up fast once Docker images and logs land |
| Access | Your SSH public key added at deploy time, so you can log in without a password |
The portal walkthrough is already documented ā follow these and come back:
\n\nOnce it's running, SSH in:
\nssh ubuntu@<your-vm-ip>
Before installing anything, confirm what you're working with ā the OS version and\nthe resources available. Note these numbers; they're useful when you later compare\nperformance or open a support ticket.
\nlsb_release -a
nproc
free -h
df -h /
On our WEC Instance: Ubuntu 22.04.5 LTS, 8 vCPU, ~15 GiB RAM, 25 GB free disk.
![\"Terminal](\"https://development-wec.wiline.com/docs/assets/images/step2-box-specs-c821280774bf4f5ec9205092f6646004.png\")
Check whether Docker is already present:
\ndocker --version
docker compose version
If it's not installed, follow Docker's official guide for your distro ā\nInstall Docker Engine on Ubuntu.\nThe quickest path is Docker's\nconvenience script,\nfollowed by adding your user to the docker group so you can run it without\nsudo:
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
You need Docker Compose v2 (the docker compose subcommand, not the legacy\ndocker-compose binary).
If Docker is already present, the get.docker.com script detects it and warns\nrather than reinstalling ā don't force it on a box with running containers. On a\nclean VM it installs from scratch. This deploy ran on a host where Docker\n29.1.3 / Compose v5.0.2 was already installed.
Create a project directory and pull OpenClaw's official docker-compose.yml:
mkdir openclaw && cd openclaw
curl -fsSL https://raw.githubusercontent.com/openclaw/openclaw/main/docker-compose.yml -o docker-compose.yml
The file defines two services ā openclaw-gateway (the long-running agent\ngateway) and openclaw-cli (a one-shot CLI that shares the gateway's network).\nBy default it builds from source; we'll point it at the prebuilt image\ninstead.
.envāCreate a .env next to the compose file. Pin the prebuilt image and set a\ndashboard token:
OPENCLAW_IMAGE=ghcr.io/openclaw/openclaw:latest
OPENCLAW_GATEWAY_TOKEN=<paste output of: openssl rand -hex 32>
.env out of gitIt holds your gateway token (and any keys). Add it to .gitignore. Commit a\n.env.example with placeholder values instead.
Pull the image:
\ndocker compose pull
OpenClaw bind-mounts ~/.openclaw into the container. Create it as your own\nuser first ā otherwise Docker creates it as root and the container (which\nruns as user node, uid 1000) can't write to it:
mkdir -p ~/.openclaw ~/.openclaw-auth-profile-secrets
If you skip this, you'll hit a permission error on first run ā see\nTroubleshooting.
\nThis is where you bring your own model ā onboarding prompts for your provider and\nAPI key. This guide uses OpenAI, but Anthropic and local models work the same way.
\nRun OpenClaw's onboarding through the gateway container. It's interactive ā it\nasks for your provider and API key and writes the config:
\ndocker compose run --rm --no-deps --entrypoint node openclaw-gateway \\
dist/index.js onboard --mode local --no-install-daemon
Walk through the prompts:
\nopenai/gpt-5.5)skip ā Skip for now (messaging channels are a later\nguide)openclaw configure)It ends with Onboarding complete.
Onboarding wrote your provider and model. Now set the gateway's runtime options ā\nrun mode, network bind, and the Control UI origins it will accept ā then start it\nas a background service.
\ndocker compose run --rm --no-deps --entrypoint node openclaw-gateway \\
dist/index.js config set --batch-json '[{\"path\":\"gateway.mode\",\"value\":\"local\"},{\"path\":\"gateway.bind\",\"value\":\"lan\"},{\"path\":\"gateway.controlUi.allowedOrigins\",\"value\":[\"http://localhost:18789\",\"http://127.0.0.1:18789\"]}]'
docker compose up -d openclaw-gateway
Container health:
\ndocker compose ps
You want Up ... (healthy). Then hit the health endpoint:
curl http://127.0.0.1:18789/healthz
# {\"ok\":true,\"status\":\"live\"}
![\"docker](\"https://development-wec.wiline.com/docs/assets/images/step9-health-2a25a28bc6b53dd20c385259373d72a5.png\")
Confirm your provider key is usable:
\ndocker compose run --rm openclaw-cli models status
# openai ... api_key=1 ... status=usable
models status prints a masked key prefix (sk-proj-ā¦). Crop or blur it before\npublishing the screenshot.
Now the real end-to-end test. The agent command needs a target, so list agents\nfirst, then message the default one (main):
docker compose run --rm openclaw-cli agents list
# - main (default)
docker compose run --rm openclaw-cli agent --agent main \\
--message \"Reply with exactly one word: working\"
# working
If it replies working, the full chain ā gateway ā OpenAI ā response ā is live.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/step9-agent-working-91514c6b42235714c1348d5ce9f6e9b9.png\")
The Control UI is at http://<your-instance-ip>:18789/; paste your\nOPENCLAW_GATEWAY_TOKEN into Settings to log in.
Browsers block the Control UI over plain HTTP at a remote IP (\"Secure browser\ncontext required\"). Reach it over an SSH tunnel so it loads as 127.0.0.1 ā see\nTroubleshooting #3.
![\"OpenClaw](\"https://development-wec.wiline.com/docs/assets/images/step9-control-ui-fe1df8b0971ad1a8192ff40d2ea6856e.png\")
These are errors we actually hit during this deploy ā not hypotheticals.
\nEACCES: permission denied, mkdir '/home/node/.openclaw/state'āHit on the first onboarding run when ~/.openclaw didn't pre-exist:
[openclaw] The CLI command failed.
[openclaw] Reason: Failed to open the plugin state database.
| EACCES: permission denied, mkdir '/home/node/.openclaw/state' | EACCES
Cause: Docker auto-created the bind-mount source ~/.openclaw owned by\nroot (0:0). The container runs as user node (uid 1000), so it can't write\ninside that root-owned directory.
Fix: give the directory to uid 1000 (which is also your ubuntu user):
sudo chown -R 1000:1000 ~/.openclaw
Pre-creating the directory yourself (Step 6) avoids this entirely.
\n![\"EACCES](\"https://development-wec.wiline.com/docs/assets/images/error1-eacces-fbbf355224203ecfccd68552fcf0bcb3.png\")
No target session selectedāThe first time we ran agent without specifying who to talk to:
Error: No target session selected. Use --agent <id>, --session-key <key>,
--session-id <id>, or --to <E.164>. Run openclaw agents list to see agents.
Cause: openclaw agent runs one turn against a specific agent/session; with\nno channel configured there's no implicit target.
Fix: list agents and pass the id explicitly:
\ndocker compose run --rm openclaw-cli agents list # shows: main (default)
docker compose run --rm openclaw-cli agent --agent main --message \"...\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/error2-no-session-c405ca89ca87bd411080165c514046e5.png\")
Opening the Control UI from another machine (e.g. macOS) at the VM's IP over\nplain HTTP, the page refuses to connect:
\nSecure browser context required
This page is running over plain HTTP, so the browser cannot create the
device identity the Gateway expects.
Cause: browsers only expose the crypto APIs OpenClaw needs in a secure\ncontext ā HTTPS, or localhost/127.0.0.1. A remote http://<ip>:18789 is\nneither, so it's blocked in the browser, regardless of the server-side\ngateway.controlUi.allowInsecureAuth flag.
Fix: tunnel the port over SSH so the browser talks to 127.0.0.1 (a secure\ncontext). On your local machine:
ssh -L 18789:127.0.0.1:18789 ubuntu@<your-vm-ip>
Then open http://127.0.0.1:18789/#token=<your-token> locally. The proper\nlong-term fix is HTTPS via a reverse proxy ā the next guide.
![\"The](\"https://development-wec.wiline.com/docs/assets/images/error3-secure-context-3e5792a56fae6a2ea2a8f9e0f0aa4c3f.png\")
On startup the gateway logged warnings worth acting on before any public\nexposure:
\ngateway.controlUi.allowInsecureAuth=true ā flagged dangerous. Run\ndocker compose run --rm openclaw-cli security audit.plugins.allow is empty ā non-bundled plugins (e.g. codex) may\nauto-load. Set an explicit allowlist of trusted plugin ids.Hardening this behind a reverse proxy with HTTPS is the subject of the next\nguide.
\nYou can now deploy and operate your own AI agent on a box you control.
You went from a bare WEC Instance to a working, self-hosted AI agent:
\nThe agent is yours: your key, your box, your data. From here it's about making it\nsecure, reachable, and useful.
\nOpenClaw isn't a chatbot ā it's an autonomous agent that can act on your box. Now\nthat it's running, here's what it unlocks (some features need extra configuration, and\nwe cover the big ones later in this series):
\nTypical real uses people run: inbox and calendar management, document processing,\ninfrastructure tasks on the box, content pipelines, and autonomous code testing.
\nThis is Part 1 of the Self-hosting OpenClaw series on running production-grade\nAI infrastructure on a WEC Instance. Coming up:
\nallowInsecureAuth workaround and reach the Control UI from anywhere, safely.Companion files for this guide ā docker-compose.yml and .env.example ā live in\nthe WiLine manifests repo (link coming with the repo).
docker compose down # stop + remove containers
docker compose down --rmi all -v # also remove the image and volumes
To remove config/state on the host:\nrm -rf ~/.openclaw ~/.openclaw-auth-profile-secrets. To stop billing entirely,\ndelete the VM from the WiLine portal.