Part 6 put required_scopes=["mcp:invoke"] on the JWT verifier and called the server authorised. It is, in the sense that an unauthenticated caller gets nothing. It is not, in the sense that matters: mcp:invoke opens every tool on the server. The same token that lets an agent run find_customer lets it run issue_refund.

That is the whole gap. A read-only agent and a refund-issuing agent hold identical credentials, and the only thing standing between "look up Maria" and "refund invoice 2" is that nobody asked.

This post closes it, badly first and then properly, because the badly is what most people ship and the difference only shows up in what the client can do about a refusal.

Both versions refuse the same call — a read-only token asking for issue_refund. They differ in where the scope is checked, and that one choice decides what the client gets back:

Prerequisites​

Parts 5 and 6, specifically:

• Authentik issuing client-credentials tokens to an agent-tools provider (part 5)
• A FastMCP server verifying those tokens against the JWKS endpoint (part 6)
• ~/mcp-auth/token.sh, which takes a scope string and returns an access token

Part 6's server stays on :8770 throughout. The two versions below run on :8771 and :8772 so you can compare all three without stopping anything.

Step 1 — Two new scopes​

Scopes are Property Mappings in Authentik. Customization → Property Mappings → New Property Mapping → Scope Mapping, twice:

Creating them is not enough. A provider will only issue a scope it has been given, so open Applications → Providers → agent-tools → Edit and move both into Selected Scopes.

Note the line under the picker: "Select which scopes can be used by the client. The client still has to specify the scope to access the data." Both halves matter. Selecting a scope here does not put it in every token — it permits the client to ask. A client that asks for nothing gets nothing, which is why every token.sh call below passes an explicit scope string.

Confirm the token actually carries what you asked for:

~/mcp-auth/token.sh "mcp:invoke office:read" \
  | cut -d. -f2 | base64 -d 2>/dev/null | jq .scope

"mcp:invoke office:read"

If that comes back without office:read, the scope is not on the provider — fix that before writing any server code, or you will spend an hour debugging enforcement that is working correctly on a token that was never scoped.

Step 2 — The obvious implementation​

Keep required_scopes=["mcp:invoke"] on the verifier as the price of admission, then ask per tool whether the caller holds what that operation needs. FastMCP exposes the verified token through get_access_token(), so a decorator can read the claims the verifier already checked:

from fastmcp.exceptions import ToolError
from fastmcp.server.dependencies import get_access_token

def requires(scope: str):
    def decorate(fn):
        @functools.wraps(fn)
        async def wrapper(*args, **kwargs):
            token = get_access_token()
            held = set(getattr(token, "scopes", None) or [])
            if scope not in held:
                raise ToolError(
                    f'insufficient_scope: this tool requires "{scope}"; '
                    f'token carries {sorted(held) or "nothing"}'
                )
            return await fn(*args, **kwargs)
        return wrapper
    return decorate

Then one line per tool:

@mcp.tool
@requires("office:read")
async def find_customer(query: str) -> list[dict]: ...

@mcp.tool
@requires("office:refund")
async def issue_refund(invoice_id: int) -> str: ...

Run it on :8771 and drive both tools with both tokens:

--- token: mcp:invoke + office:read ---
  find_customer  : ALLOWED -> [{'id': 1, 'name': 'Maria Alvarez', ...}]
  issue_refund   : REFUSED -> insufficient_scope: this tool requires "office:refund"

--- token: mcp:invoke + office:refund ---
  find_customer  : REFUSED -> insufficient_scope: this tool requires "office:read"
  issue_refund   : ALLOWED -> Refunded 80.00 on invoice 2.

That is real enforcement. The refund token cannot read, the read token cannot refund, and the refusal names the missing scope. For a lot of deployments this is where you stop.

Step 3 — Why that refusal is not good enough​

Watch the HTTP layer rather than the client library.

TOK=$(~/mcp-auth/token.sh "mcp:invoke office:read")
curl -sS -i -X POST http://127.0.0.1:8771/mcp \
  -H "Authorization: Bearer $TOK" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"issue_refund","arguments":{"invoice_id":2}}}'

HTTP/1.1 200 OK
content-type: text/event-stream

data: {"jsonrpc":"2.0","id":1,"result":{"content":[{"text":"insufficient_scope: this tool
requires \"office:refund\"; token carries ['mcp:invoke', 'office:read']","type":"text"},
"isError":true}}

HTTP 200. The request succeeded; the tool declined. That is correct JSON-RPC semantics and it is useless to an OAuth client.

An OAuth client that wants to step up — go back to the authorization server and ask for office:refund — is watching for a 401 or 403 with a WWW-Authenticate header telling it what to request. It does not parse English out of a tool result. So the refusal is legible to a human reading logs and invisible to the machinery designed to handle exactly this case.

It also arrives late. The tool body runs after routing, after session setup, after the server has committed to a successful response. By then there is no status line left to change.

Step 4 — The challenge the spec actually wants​

Getting a 403 means checking scopes before the JSON-RPC layer answers, which means ASGI middleware wrapping the FastMCP app:

class ScopeChallengeMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http" or scope["method"] != "POST":
            return await self.app(scope, receive, send)

        # Buffer the body so we can inspect it and still pass it on.
        chunks, more = [], True
        while more:
            msg = await receive()
            chunks.append(msg.get("body", b""))
            more = msg.get("more_body", False)
        body = b"".join(chunks)

        needed = None
        try:
            rpc = json.loads(body)
            if rpc.get("method") == "tools/call":
                needed = TOOL_SCOPES.get((rpc.get("params") or {}).get("name"))
        except Exception:
            pass

        if needed and needed not in token_scopes(dict(scope.get("headers") or [])):
            challenge = (
                f'Bearer error="insufficient_scope", scope="{needed}", '
                f'resource_metadata="{RESOURCE_METADATA}", '
                f'error_description="This operation requires the {needed} scope"'
            )
            await send({"type": "http.response.start", "status": 403,
                        "headers": [(b"content-type", b"application/json"),
                                    (b"www-authenticate", challenge.encode())]})
            await send({"type": "http.response.body",
                        "body": json.dumps({"error": "insufficient_scope",
                                            "scope": needed}).encode()})
            return

        # Replay the buffered body downstream.
        replayed = False
        async def replay():
            nonlocal replayed
            if not replayed:
                replayed = True
                return {"type": "http.request", "body": body, "more_body": False}
            return await receive()

        await self.app(scope, replay, send)

app = ScopeChallengeMiddleware(mcp.http_app(stateless_http=True))

Same request, against :8772:

HTTP/1.1 403 Forbidden
www-authenticate: Bearer error="insufficient_scope", scope="office:refund",
  resource_metadata="http://127.0.0.1:8772/.well-known/oauth-protected-resource",
  error_description="This operation requires the office:refund scope"

Now a client has something to act on: the status says refused, scope= says what to ask for, and resource_metadata says where to look up how. That is step-up authorization as a protocol rather than as a log message.

Step 5 — What it cost​

The middleware works. It is also worse code than the decorator, in three specific ways, and pretending otherwise would be dishonest.

It does not know what a tool is. ASGI middleware sees bytes and headers. To find out which tool is being called it parses the JSON-RPC envelope itself and looks the name up in a table it has to maintain:

TOOL_SCOPES = {
    "find_customer": "office:read",
    "open_invoices": "office:read",
    "issue_refund":  "office:refund",
}

That table is a second source of truth. Add a tool and forget the entry and it is unprotected — silently, because the middleware just passes through anything it does not recognise. The decorator could not have that bug: the requirement sat on the function.

It decodes the token unverified. The middleware runs upstream of the verifier, so the verified claims do not exist yet. It splits the JWT and base64-decodes the payload without checking the signature:

payload = auth.split(None, 1)[1].split(".")[1]
claims = json.loads(base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4)))

This is not the hole it looks like — the verifier still runs downstream and still rejects a forged token, so nothing reaches a tool on a bad signature. But the scope decision is made on unauthenticated bytes, and the only reason that is survivable is the second check behind it. It is a wart, not a vulnerability, and it is the kind of thing worth writing down before someone later removes the "redundant" verifier.

It buffers every request body. To read the envelope and still pass it downstream, the middleware drains receive() into memory and replays it. Fine for JSON-RPC calls; think harder before putting this in front of large uploads.

Troubleshooting — the errors this run actually produced​

insufficient_scope on a tool you did grant​

Check the token, not the server:

~/mcp-auth/token.sh "mcp:invoke office:refund" | cut -d. -f2 | base64 -d 2>/dev/null | jq .scope

If office:refund is missing, the scope exists as a Property Mapping but was never moved into the provider's Selected Scopes. Authentik silently drops scopes a client is not permitted to request rather than erroring, so the token comes back valid and short.

The middleware never fires​

It only inspects POST. MCP clients open a GET for the event stream first, and that request carries no JSON-RPC envelope — if you are watching the wrong request you will conclude the middleware is dead. Confirm with the curl above, which is a single POST.

Every tool suddenly returns 403​

TOOL_SCOPES is keyed by the tool's registered name, which is the function name, not the decorated label. Rename a function and the table stops matching. The pass-through case is the dangerous direction (unprotected), but a typo in the table produces this one.

The 403 body is empty in some clients​

The challenge lives in the WWW-Authenticate header. Clients that only log response bodies will show you {"error":"insufficient_scope"} and nothing about which scope. Use curl -i.

What this did and didn't buy you​

It bought genuine per-tool authorisation: two tokens that differ by one scope, each able to run exactly one of two tools, proven at the HTTP layer rather than asserted. And with the middleware, a refusal a client can programmatically recover from.

It did not buy a clean design. Both implementations are compromises pointing opposite ways:

The honest summary is that the correct protocol behaviour requires leaving the abstraction the framework gives you, and the ergonomic version cannot produce it. If your clients do not implement step-up — and most agent clients today do not — the decorator is the better trade. If they do, you pay for it with a table you must remember to update.

It also did not buy authorisation that survives the tool doing something else. issue_refund is gated on office:refund; nothing stops a future find_customer from being edited to write. Scopes gate entry, not behaviour — which is the same boundary part 4 drew around policy-in-the-tool.

What's next​

The obvious remaining gap is that both versions trust the token's scope list and nothing else. Neither asks who the caller is or what they are acting on — a token with office:refund refunds any invoice, for any customer, for any amount. That is object-level authorisation, and it does not live in OAuth scopes at all.

Further reading​

• MCP specification — authorization
• RFC 6750 §3 — the WWW-Authenticate challenge
• RFC 9728 — OAuth 2.0 Protected Resource Metadata
• Authentik — property mappings and scopes

Every agent tutorial in this series so far has given a model tools — functions you wrote, with arguments you defined. This post is about the other thing agents do, which is write code and then run it.

That is a different risk, and the difference is worth being precise about. A tool call is the model choosing from a menu you control. Executing generated code is the model handing you something nobody has ever read, which you then run on your machine.

Why this needs its own post​

The industry has a name for this. OWASP's LLM05:2025 Improper Output Handling lists it as a vulnerability example, in plain words:

LLM output is entered directly into a system shell or similar function such as exec or eval, resulting in remote code execution.

And its advice on how to think about the model is exactly right:

Treat the model as any other user, adopting a zero-trust approach

Here is what makes that page worth reading closely, though. It tells you to treat model output as untrusted, it names code execution as the consequence, and then its prevention section talks entirely about validating text — output encoding, parameterised queries, content security policies, monitoring.

Search that page for sandbox, container, least privilege or isolation and you get zero hits. Not few. None.

That is not a criticism of OWASP; the risk is about output handling, and validation is the right answer for output that becomes HTML or SQL. But when the output is a program you are about to run, validation stops being possible. You cannot regex your way to knowing whether a hundred lines of Python are safe.

So the guidance names the problem and stops one layer above the fix. This post is that layer.

What the obvious version actually allows​

The naive implementation is what most people ship first, and it looks responsible — the code runs in a container, after all:

docker -c rootless run --rm python:3.12-alpine python -c "<the model's code>"

Let's ask that container what it can reach.

docker -c rootless run --rm python:3.12-alpine python -c "
import os, urllib.request
print('uid inside     :', os.getuid())
print('can see host / :', sorted(os.listdir('/'))[:10])
try:
    r = urllib.request.urlopen('https://api.github.com', timeout=5)
    print('internet       : REACHED', r.status)
except Exception as e:
    print('internet       : blocked', type(e).__name__)
"

docker -c rootless run --rm python:3.12-alpine sh -c '
echo "memory limit : $(cat /sys/fs/cgroup/memory.max)"
echo "pids limit   : $(cat /sys/fs/cgroup/pids.max)"
echo "cpus visible : $(nproc)"
echo "writable fs  : $(touch /proc-test 2>/dev/null && echo yes || echo no)"
'

internet       : REACHED 200
memory limit : max
pids limit   : 19136
cpus visible : 8
writable fs  : yes

Read that as a list of what generated code is allowed to do on your machine:

• Reach the internet. Anything it finds, it can send somewhere.
• Take all the memory. max means no limit. On this host that is 15 GB, and the kernel will start killing other things to provide it.
• Start 19,136 processes. Enough to make the host unusable.
• Use all eight cores.
• Write anywhere in the container, including filling the disk.

Part 8 moved the daemon off root, so this code can no longer take over the host. That was worth doing and it does not help here at all. None of the five items above involves being root.

Closing it down​

Docker has flags for every one of these. They are unglamorous and they are the entire fix:

docker -c rootless run --rm \
  --network none --memory 256m --pids-limit 64 --cpus 0.5 \
  --read-only --tmpfs /tmp:size=16m \
  --cap-drop=ALL --security-opt no-new-privileges \
  python:3.12-alpine <command>

In plain terms:

Run it, and it fails before the container even starts:

docker: Error response from daemon: NanoCPUs can not be set, as your kernel does not
support CPU CFS scheduler or the cgroup is not mounted

Hold onto that message, because it is misleading. The kernel supports CPU limits perfectly well — this host runs 20 other containers with them available. We will come back to what is actually wrong.

Drop --cpus and everything else applies:

memory limit : 268435456        <- exactly 256 MiB
pids limit   : 64
cpus visible : 8
writable fs  : no
tmp writable : yes
internet     : blocked URLError

Five of six controls working. And cpus visible: 8, because we were not allowed to set one.

Proving the gap is real​

A missing CPU limit sounds mild next to "unlimited memory". It is not, because an infinite loop is the single most likely thing generated code does by accident.

This runs eight busy loops inside the otherwise fully locked-down sandbox, for eight seconds:

docker -c rootless run -d --name burn \
  --network none --memory 256m --pids-limit 64 \
  --read-only --tmpfs /tmp:size=16m \
  --cap-drop=ALL --security-opt no-new-privileges \
  python:3.12-alpine sh -c 'for i in $(seq 8); do (while :; do :; done) & done; sleep 8'
sleep 3
docker -c rootless stats --no-stream burn

NAME   CPU %     MEM USAGE / LIMIT   NET I/O   PIDS
burn   605.19%   1.23MiB / 256MiB    0B / 0B   9

605% of a CPU. Six of the host's eight cores, taken by code inside a sandbox where memory is capped at 256 MB (it used 1.23), processes are capped at 64 (it used 9), and the network is completely blocked.

Every control we set is working. The one we could not set is the one being abused.

What is actually wrong​

The error message blamed the kernel. The kernel is fine. The real answer is one file.

When Docker runs rootless, it does not get to manage resources directly — it can only use what the system has handed to your user account. On modern Linux that handover is called cgroup delegation, and systemd decides what gets delegated.

cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers

memory pids

Two things. That is the whole explanation:

The daemon knew all along. It said so at startup, ten times, in lines nobody reads:

journalctl --user -u docker.service | grep -i "no .* support"

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.max (rbps) support
...

This also explains something Part 8 reported as a plain limitation of rootless mode: disk throughput limits not working. Same cause. Not a property of rootless — a property of what systemd delegated.

Where this is documented​

Not in Docker's rootless documentation. I searched the page: zero occurrences of "delegate". The guidance lives upstream, on the rootless containers site, which states it directly:

By default, a non-root user can only get memory controller and pids controller to be delegated.

The fix​

One drop-in file telling systemd to hand over more:

sudo mkdir -p /etc/systemd/system/user@.service.d
cat <<EOF | sudo tee /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=cpu cpuset io memory pids
EOF
sudo systemctl daemon-reload

The same page warns:

After changing the systemd configuration, you need to re-login or reboot the host. Rebooting the host is recommended.

On systemd 249 that turned out not to be needed. After daemon-reload alone:

cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers

cpuset cpu io memory pids

The daemon still needs restarting, because it reads its capabilities once at startup. Restart only the rootless daemon — not your whole user session, which on this host would also have taken down an unrelated service:

systemctl --user restart docker.service
journalctl --user -u docker.service --since "-1min" | grep -ci "no cpu\|no io"

0

Zero warnings, where there were ten.

The same eight loops, again​

Identical command to before. One flag added:

docker -c rootless run -d --name burn2 \
  --network none --memory 256m --pids-limit 64 --cpus 0.5 \
  --read-only --tmpfs /tmp:size=16m \
  --cap-drop=ALL --security-opt no-new-privileges \
  python:3.12-alpine sh -c 'for i in $(seq 8); do (while :; do :; done) & done; sleep 8'
sleep 3
docker -c rootless stats --no-stream burn2

NAME    CPU %    MEM USAGE / LIMIT    NET I/O   PIDS
burn2   52.82%   1.242MiB / 256MiB    0B / 0B   9

605.19% → 52.82%. Same eight infinite loops, same image, same everything. Half a core, as asked.

What you should take from this​

A container is not a sandbox until you say what it may use. The default is generous in every dimension that matters, and nothing warns you, because an unrestricted container is a completely normal thing to run.

Check your limits are applied, rather than assuming. The whole of this post exists because a flag that was silently unavailable looked identical to a flag that was working. docker stats under a deliberate load takes a minute and tells you the truth.

Error messages point at the wrong layer. "your kernel does not support CPU CFS scheduler" described a kernel that supports it fine. The real cause was a systemd default, two layers away, documented on a different website.

And the honest limit: this sandbox contains resources. It does not make the code safe. It cannot tell a useful script from a hostile one, and a program that does something harmful within 256 MB, half a core and no network will run exactly as written. Containment buys you a blast radius, not a judgement.

Further reading​

• OWASP — LLM05:2025 Improper Output Handling
• Docker — Runtime options with Memory, CPUs, and GPUs
• Rootless Containers — cgroup v2 delegation
• Linux kernel — Control Group v2
• systemd.resource-control(5) — Delegate=

Here is the part nobody says out loud when they tell you to add yourself to the docker group so you can stop typing sudo.

That group is root. Not "close to root", not "root for Docker things". If you can run a container, you can read, change or delete any file on the machine — including the password file, including other people's home directories, including the files the administrator deliberately kept away from you.

No exploit required. It is one command, and it takes about four seconds.

Proving it on your own machine​

Run this as a normal user. There is no sudo anywhere in it.

docker run --rm -v /:/host alpine:3.20 \
  sh -c 'touch /host/root/rootless-demo && echo WROTE as uid $(id -u)'
sudo ls -l /root/rootless-demo

WROTE as uid 0
-rw-r--r-- 1 root root 0 Sep 16 20:58 /root/rootless-demo

A user who cannot so much as list /root just created a file in it, owned by root.

Nothing malfunctioned. That is the design. The daemon runs as root, you asked it to mount the whole filesystem, and it did — because a root process is allowed to. The container's "root" is the host's root. They are the same thing.

Docker documents this plainly, and it is worth reading their wording rather than mine. From Docker daemon attack surface:

only trusted users should be allowed to control your Docker daemon

and, on the same page:

Docker allows you to share a directory between the Docker host and a guest container; and it allows you to do so without limiting the access rights of the container.

That is not a warning about a flaw. It is a description of the feature we just used.

Clean it up before moving on:

sudo rm -f /root/rootless-demo

Why this is worse on a host running agents​

An AI agent that executes code it generated is, in security terms, remote code execution that you installed on purpose. That is not a criticism of agents — it is what they are for.

So the question stops being "could an attacker run code here?" — something already does, constantly — and becomes "what can that code reach?"

If it can reach the Docker socket, the answer is everything, via the command above. Your API keys. Your SSH keys. The database volume. All of it, regardless of how carefully the agent's own container was configured.

What Part 2 did and did not fix​

Part 2 found two of six Langfuse containers running as root and fixed them with user: "999:999". That was worth doing and it is still worth doing.

But it changes who the process is inside the container. The daemon that starts it is still root, and the socket is still a door to the whole machine. Part 2 hardened the passenger. This post goes after the driver.

What rootless mode actually does​

The daemon stops running as root and runs as you.

Containers still think they have root — software inside them keeps working normally. But that "root" is now translated, by the kernel, into an ordinary unprivileged account that owns nothing important. Escape the container and you are not the administrator. You are ubuntu.

The translation uses a block of user IDs the system set aside for you. Here is the whole idea in one picture — the same container, under each daemon:

The left half is why the first command worked. The right half is what we are about to build, and the arithmetic in it is something you can verify yourself in a minute.

This translation is a Linux kernel feature called a user namespace, not a Docker invention — Docker is just asking the kernel for one. The block of IDs it maps into is declared in /etc/subuid:

grep '^ubuntu:' /etc/subuid /etc/subgid

/etc/subuid:ubuntu:100000:65536
/etc/subgid:ubuntu:100000:65536

65,536 IDs starting at 100000, yours alone. Hold onto that number — the mapping becomes checkable arithmetic in a moment.

Installing it: four walls​

The documentation assumes a setup this machine does not have. Here is each wall in the order you hit it.

Wall 1 — the package does not exist​

Every guide opens with the same line. On Ubuntu's Docker:

sudo apt-get install -y docker-ce-rootless-extras
dpkg -L docker.io | grep -iE 'rootless|rootlesskit'

E: Unable to locate package docker-ce-rootless-extras

That package belongs to Docker's own repository. This host runs Ubuntu's docker.io, and the second command returns nothing at all — the distribution package ships no rootless tooling whatsoever.

That matters because it forces a choice. You can replace docker.io with Docker CE, which means swapping out the daemon currently running your containers. Or you can install a second, user-owned daemon beside the first. With 20 services running, the second option is the only sane one, and it is what the rest of this post does.

What you do need from apt is the ID-mapping helpers:

sudo apt-get install -y uidmap

Wall 2 — the installer refuses to run​

curl -fsSL https://get.docker.com/rootless | sh

# Installing stable version 29.8.1
Aborting because rootful Docker is running and accessible.
Set FORCE_ROOTLESS_INSTALL=1 to ignore.

The guard assumes you are migrating — that a running root daemon means you are about to make a mess. Running both side by side on purpose is never mentioned, and the way to proceed appears only inside the error text.

Note the version it names: 29.8.1, while the system daemon is 29.1.3. You will end up with two different Docker versions on one host. That is expected here, and worth knowing.

curl -fsSL https://get.docker.com/rootless | FORCE_ROOTLESS_INSTALL=1 sh

Wall 3 — it moves your CLI without telling you​

Read the last lines of the installer output:

[INFO] Creating CLI context "rootless"
[INFO] Using CLI context "rootless"
Current context is now "rootless"

Now ask Docker what is running:

docker ps
docker context ls
docker -c default ps -q | wc -l

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

NAME         DESCRIPTION                              DOCKER ENDPOINT                     ERROR
default      Current DOCKER_HOST based configuration  unix:///var/run/docker.sock
rootless *   Rootless mode                            unix:///run/user/1000/docker.sock

20

An empty list, while twenty containers run perfectly well. Nothing broke and nothing warned you — the CLI is answering a different question than the one you think you asked, because the installer repointed it on the way out.

Put it back, and opt in per command instead:

docker context use default          # everyday work, unchanged
docker -c rootless ps               # explicit when you want the new one

Start the daemon and confirm what you have:

export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user start docker.service
docker -c rootless info --format '{{.ServerVersion}} {{.SecurityOptions}}'

29.8.1 [name=seccomp,profile=builtin name=rootless name=cgroupns]

name=rootless is the daemon confirming what it is.

Checking that the mapping is real​

Ask a container who it is, then look at that same process from outside:

docker -c rootless run --rm alpine:3.20 id
docker -c rootless run -d --name u0 alpine:3.20 sleep 60
ps -o user=,uid=,cmd= -C sleep | grep 'sleep 60'

docker -c rootless run -d --name u1000 --user 1000:1000 alpine:3.20 sleep 120
ps -o uid=,cmd= -C sleep | grep 'sleep 120'

uid=0(root) gid=0(root) groups=0(root),...
ubuntu    1000 sleep 60
100999 sleep 120

Read those three lines together, because they are the entire mechanism:

• The container believes it is root.
• The host says that process belongs to ubuntu.
• A container running as ID 1000 shows up on the host as 100999.

That last number is 100000 + 1000 - 1 — the start of your reserved block, offset by the container's own ID, because container root already took the first slot by mapping to you.

Which gives you a test rather than a promise. Work it out from your own /etc/subuid line and see whether the number matches. If you instead see plain 1000, the translation is not happening and something is wrong with the setup.

Clean up:

docker -c rootless rm -f u0 u1000

The same command, the other daemon​

Back to the command this post opened with. Same image, same mount of the entire filesystem, same flags. Only the daemon differs:

docker -c rootless run --rm -v /:/host alpine:3.20 \
  sh -c 'touch /host/root/rootless-demo || echo DENIED'

touch: /host/root/rootless-demo: Permission denied
DENIED

The container still believes it is root. The kernel disagrees, because the process behind it belongs to ubuntu, and ubuntu cannot write to /root.

That is the post, in two outputs.

What it costs you​

Four things, all measured here rather than quoted.

Containers do not start noticeably slower. Three consecutive runs:

for i in 1 2 3; do /usr/bin/time -f "run+exit: %es" \
  docker -c rootless run --rm alpine:3.20 true; done
docker -c rootless run --rm -p 80:80 alpine:3.20 true

run+exit: 1.19s
run+exit: 0.96s
run+exit: 0.97s

Ports below 1024 are refused, and unusually, the error tells you every way out:

cannot expose privileged port 80, you can add
'net.ipv4.ip_unprivileged_port_start=80' to /etc/sysctl.conf (currently 1024),
or set CAP_NET_BIND_SERVICE on rootlesskit binary,
or choose a larger port number (>= 1024)

-p 18080:80 binds without complaint. Behind a reverse proxy this rarely matters.

Disk throughput limits stop working. The daemon says so at startup, and it is easy to miss:

WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support

So --device-read-bps and --device-write-bps silently do nothing. If you were relying on those to stop one container starving the others, rootless takes that away.

Capability dropping still works, which matters because these are often treated as alternatives:

docker -c rootless run --rm --cap-drop=ALL alpine:3.20 sh -c 'echo cap-drop ok'

cap-drop ok

They stack. Keep using user: and --cap-drop=ALL — each layer assumes the one before it failed.

Shared folders behave differently​

This is the part that catches people moving a real service across. Host permissions apply as you, not as root:

D=/tmp/rl-vol; mkdir -p $D; echo hello > $D/mine.txt
sudo sh -c "echo secret > $D/theirs.txt; chmod 600 $D/theirs.txt"
docker -c rootless run --rm -v $D:/data alpine:3.20 sh -c \
  'cat /data/mine.txt; cat /data/theirs.txt 2>&1 || echo DENIED'
docker -c rootless run --rm -v $D:/data alpine:3.20 stat -c '%u:%g %n' /data/mine.txt

hello
cat: can't open '/data/theirs.txt': Permission denied
DENIED
0:0 /data/mine.txt

Your own file reads fine. The root-owned one is refused — even though the container thinks it is root, because the kernel knows better. And your file appears inside the container as owned by 0:0, the same translation running in reverse.

Worth internalising before you move a database over and spend an hour wondering why it cannot open its own data directory.

One more: the rootless daemon keeps a separate image store at ~/.local/share/docker. Images your system daemon already has are invisible to it, and everything gets pulled again. On a tight disk, budget for it.

Wall 4 — removing it refuses too​

For symmetry, the uninstall has the same guard:

PATH=/home/ubuntu/bin:$PATH ~/bin/dockerd-rootless-setuptool.sh uninstall

[ERROR] Aborting because rootful Docker (/var/run/docker.sock) is running and accessible.
Set --force to ignore.

Add --force when you genuinely want it gone.

What this does and does not buy you​

It changes the worst case from "an attacker owns the machine" to "an attacker owns this user account". On a host where every service runs as ubuntu, that is a large move and not a complete one — your SSH keys, your .env files and your source code all still sit inside that account.

So the honest summary: rootless raises the floor. It does not make the room safe. Narrowing what the account itself can reach is the next rung, and that is where system-call filtering and image supply chain come in.

But the specific thing that was true at the top of this post — that anyone able to run a container could take the whole machine — is no longer true, and you can prove it in one command.

Further reading​

• Docker — Run the daemon as a non-root user (rootless mode)
• Docker — Isolate containers with a user namespace
• rootlesskit
• subuid(5) and user_namespaces(7)

Agent orchestration part 4 split a toolbox so the scheduler could not issue refunds. Part 5 gave the agent an identity at the gateway. Part 6 put a JWT verifier in front of the tools server so it refuses anonymous callers.

Every one of those controls what the agent may call. Not one of them controls where the agent may go.

That distinction is the whole of this post. Exfiltration does not need a tool. It needs a socket.

Two directions, and only one is usually guarded​

A firewall decides which packets may cross a boundary. Most of the time people mean it in one direction — ingress, who out there may reach in to your service. That is what publishing a port does, and what part 1 of this series was about.

Egress is the other direction: where your service may reach out to. Which addresses it may open a connection to, and which it may not.

Hardly anyone configures egress, and for an ordinary web application that is a reasonable choice. It talks to its database and a payment provider, it does what its code says, and its code does not change between deployments.

An agent is not that. An agent decides what to do at runtime, from text it has just read — an email, a support ticket, a web page, a document somebody else wrote. If an attacker controls that text, they influence what the agent does next. And the thing you do not want it doing next is exfiltration: sending your data somewhere it should not go.

Exfiltration needs no special tool. Any process that can open a socket can send anything it can read. Which is why the direction nobody guards is the one that matters here.

What the standards actually say​

The leading reference for this class of problem is OWASP's LLM06:2025 Excessive Agency. Its prevention section lists, in order: minimize extensions, minimize extension functionality, avoid open-ended extensions, and further items about permissions and human approval.

Read the page looking for the network and it is not there. Not "egress", not "firewall", not "network", not "isolation", not "sandbox" — the words do not appear in the entry at all.

Every mitigation on that list operates on tools. Remove the extension, narrow the extension, avoid the open-ended extension. All good advice. None of it changes a single byte of what a compromised process can do with a TCP socket.

There is an argument that this is a gap rather than an oversight, and it is being made inside OWASP's own repository. An open issue proposing a runtime-enforcement mapping for the Agentic Top 10 states it plainly:

The enforcement pipeline assumes agents cannot reach tools without passing through the proxy. If tools are directly reachable, runtime enforcement collapses.

and concludes that "Network-level controls (e.g., Kubernetes NetworkPolicy, Docker network isolation) are required to close this gap in production."

Note what that is: an open issue, filed by a contributor, with its mapping marked (Proposed). It is not the standard. It is someone arguing the standard should say this.

And the reason this matters is not theoretical. EchoLeak (CVE-2025-32711) is a zero-click indirect prompt injection against Microsoft 365 Copilot, recorded in NVD as "Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network." Over a network. The tool list was never the thing that failed.

Prerequisites​

• Docker on Linux with the iptables firewall backend — the default. Check with docker info | grep -i "Firewall Backend"
• sudo, and a willingness to write firewall rules on a box you can still reach
• Nothing from the earlier parts. This one stands alone.

Step 1 — Prove the gap​

A network, with the bridge interface named explicitly. Docker documents the option as "Interface name to use when creating the Linux bridge":

docker network create --driver bridge \
  -o com.docker.network.bridge.name=br-agent agent-egress

ip -br link show br-agent

br-agent         DOWN           8a:77:65:54:9b:0d <NO-CARRIER,BROADCAST,MULTICAST,UP>

DOWN, because nothing is plugged into it yet. A Docker network is an interface that does not come up until a container attaches, and a rule matching -i br-agent on an idle network silently matches nothing. Worth knowing before you spend an hour debugging a policy that was never exercised.

The sandbox — pinned, not latest:

docker run -d --name agent-sandbox --network agent-egress alpine:3.20 sleep infinity

docker exec agent-sandbox apk add --no-cache curl bind-tools

Now four destinations. Note what is not in this container: no agent, no framework, no MCP client, no tools. Tool scoping is not being bypassed here — there is nothing to bypass.

docker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'inference API   HTTP %{http_code}\n' https://inference.wiline.com/v1/models
docker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'arbitrary host  HTTP %{http_code}\n' https://example.com
docker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'authentik LAN   HTTP %{http_code}\n' http://10.80.4.212:9100/
docker exec agent-sandbox curl -sS -m 8 -o /dev/null -w 'other network   exit=%{exitcode} %{errormsg}\n' http://172.21.0.2:5432/

inference API   HTTP 401
arbitrary host  HTTP 200
authentik LAN   HTTP 302
other network   exit=28 Connection timed out after 5002 milliseconds

The 401 is the useful one to understand. No API key was sent, so the request never reaches anything that could charge for it — but a 401 is an HTTP response, which means the TCP handshake completed, TLS negotiated, and a real server answered. Reachability, proven, for free.

The fourth line is the one that keeps this honest. That is a Postgres container on a different Docker network, and it times out — part 1 of this series established that separate bridge networks genuinely isolate, and this confirms it.

So the shape of the hole is precise rather than vague:

Docker isolates containers from each other and stops there. The two directions that matter for exfiltration — your network, and the internet — are wide open, and no Docker guide frames that as a gap because, for a web application, it isn't one.

Step 2 — The floor​

One rule. Note -A rather than -I: the deny goes at the bottom of the chain and stays there, and every allow added later stacks above it. That ordering is the policy.

sudo iptables -A DOCKER-USER -i br-agent -j DROP

DOCKER-USER is where this belongs, and Docker's documentation is explicit about why:

Rules appended to the FORWARD chain will be processed after Docker's rules.

DOCKER-USER is processed before them — the docs describe it as "a placeholder for user-defined rules that will be processed before rules in the DOCKER-FORWARD and DOCKER chains." Anywhere else and Docker's own ACCEPT gets there first.

Read that left to right and the placement explains itself. DOCKER-FORWARD holds one unconditional ACCEPT per bridge — that is what makes container egress work at all, and it is why a rule appended to FORWARD never fires. DOCKER-USER is the only hook that sits upstream of it.

Re-run the same four probes:

inference API   exit=28 Resolving timed out after 8000 milliseconds
arbitrary host  exit=28 Resolving timed out after 8002 milliseconds
authentik LAN   exit=28 Connection timed out after 8003 milliseconds
other network   exit=28 Connection timed out after 8002 milliseconds

Read the error text, not the failure. Two different messages, and the difference is free diagnostic information:

• "Resolving timed out" — DNS itself was blocked. We never learned an address.
• "Connection timed out" — the address was already known, the packet left, the drop caught it.

The two by-name probes died at resolution; the two by-IP probes died at the socket. Which tells us something not obvious: Docker's embedded resolver forwards its upstream queries across DOCKER-USER. The resolver lives at 127.0.0.11 inside the container's namespace, and it is tempting to assume that traffic never touches the host's forward path. It does.

Also note that every failure took the full eight seconds. A DROP produces timeouts; a REJECT fails instantly. Stealth against debuggability, and this post chooses stealth — but if you are still iterating on a policy, REJECT will save you a great deal of waiting.

Step 3 — The allowlist​

Before writing a rule for the resolver, find out which resolver the container actually uses:

docker exec agent-sandbox cat /etc/resolv.conf

nameserver 127.0.0.11
search corp.internal internal mesh.example.com
options edns0 trust-ad ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [8.8.8.8 8.8.4.4]

ExtServers: [8.8.8.8 8.8.4.4]. The container is not using this network's DNS. The host resolves through systemd-resolved on 127.0.0.53, which is a loopback address and therefore useless inside a container's namespace, so Docker substituted Google Public DNS without mentioning it.

Look at the inherited search domains in the same block (redacted here, but real on your box). Every internal hostname that container looks up is being asked of 8.8.8.8, suffixed with the private domains it inherited from the host. Nobody chose that, and nothing surfaces it except reading this file.

It also decides the rule: allow the resolver the container has, not the one you assumed.

getent hosts inference.wiline.com

67.207.107.229  inference.wiline.com

sudo iptables -I DOCKER-USER -i br-agent -p udp --dport 53 -d 8.8.8.8,8.8.4.4 -j ACCEPT
sudo iptables -I DOCKER-USER -i br-agent -p tcp --dport 53 -d 8.8.8.8,8.8.4.4 -j ACCEPT
sudo iptables -I DOCKER-USER -i br-agent -p tcp --dport 443 -d 67.207.107.229 -j ACCEPT

sudo iptables -S DOCKER-USER

-N DOCKER-USER
-A DOCKER-USER -d 67.207.107.229/32 -i br-agent -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER-USER -d 8.8.4.4/32 -i br-agent -p tcp -m tcp --dport 53 -j ACCEPT
-A DOCKER-USER -d 8.8.8.8/32 -i br-agent -p tcp -m tcp --dport 53 -j ACCEPT
-A DOCKER-USER -d 8.8.4.4/32 -i br-agent -p udp -m udp --dport 53 -j ACCEPT
-A DOCKER-USER -d 8.8.8.8/32 -i br-agent -p udp -m udp --dport 53 -j ACCEPT
-A DOCKER-USER -i br-agent -j DROP

Five allows on one deny. The same four probes, a third time:

inference API   HTTP 401 exit=0
arbitrary host  exit=28 Connection timed out after 8002 milliseconds
authentik LAN   exit=28 Connection timed out after 8003 milliseconds
other network   exit=28 Connection timed out after 8001 milliseconds

Look at line two. It says "Connection timed out", not "Resolving timed out". DNS is open, so example.com resolved to a real address — and then the packet died. The lookup succeeded and the socket did not.

That is the whole post in one line. A compromised process inside that container can discover where to send your data and cannot send it.

Step 4 — Measuring the cost, twice​

A policy nobody measures is a policy someone will disable under load. So: what does six rules cost?

The obvious approach, twenty connects with the rules on, then twenty with them off:

for i in $(seq 20); do docker exec agent-sandbox curl -sS -o /dev/null -w '%{time_connect}\n' -m 8 https://inference.wiline.com/v1/models; done | sort -n | awk '{a[NR]=$1} END{printf "rules ON  — median %.4fs  min %.4fs  max %.4fs  (n=%d)\n", a[int(NR/2)+1], a[1], a[NR], NR}'

sudo iptables -F DOCKER-USER

rules ON  — median 0.1573s  min 0.0940s  max 0.3768s  (n=20)
rules OFF — median 0.0936s  min 0.0911s  max 0.1576s  (n=20)

64 milliseconds. Which would be a serious finding, if it were true.

It is not. Six rules of packet matching is microseconds of kernel work, and the minimums give the game away: 0.0940 against 0.0911, near-identical. The fast path is the same in both runs. The entire difference lives in the tail.

curl's time_connect is measured from the start of the request and therefore includes name resolution. The first run paid a full round trip to 8.8.8.8. By the second, the resolver had the answer cached. We measured the order we ran them in.

Measure the TCP connect alone, by subtracting the lookup:

for i in $(seq 30); do docker exec agent-sandbox curl -sS -o /dev/null -w '%{time_namelookup} %{time_connect}\n' -m 8 https://inference.wiline.com/v1/models; done | awk '{print ($2-$1)}' | sort -n | awk '{a[NR]=$1} END{printf "connect-only median %.5fs  min %.5fs  max %.5fs  (n=%d)\n", a[int(NR/2)+1], a[1], a[NR], NR}'

rules ON  — connect-only median 0.07098s  min 0.06999s  max 0.08607s  (n=30)
rules OFF — connect-only median 0.07102s  min 0.06996s  max 0.07214s  (n=30)

A median difference of −0.04 ms. Negative: the firewalled path measured marginally faster, which is what noise looks like. Everything sits inside about ±40 µs.

So the honest answer is not a number with a plus sign in front of it. It is that the cost of this policy is below what this method can resolve — and that the naive version of the same measurement was wrong by three orders of magnitude, in the direction that would have talked you out of deploying it.

Step 5 — Surviving a reboot​

iptables rules live in memory. That -F in the last step did to the policy exactly what a reboot does.

The reflex is iptables-persistent, and it is the wrong tool here: it saves the whole table including every rule Docker generated, and Docker regenerates those itself at start. You get duplicates and an ordering nobody chose.

What you want is a unit that owns only your rules and runs after Docker:

#!/usr/bin/env bash
# Egress allowlist for the agent bridge. Re-applied after Docker starts.
set -euo pipefail

BR=br-agent
DNS="8.8.8.8 8.8.4.4"
API=$(getent hosts inference.wiline.com | awk '{print $1; exit}')

# Remove only our own rules, so this is safe to run twice.
# `|| true` matters: on a clean chain grep exits 1 and would abort the script.
existing=$(iptables -S DOCKER-USER | grep " -i ${BR} " || true)
if [ -n "${existing}" ]; then
  printf '%s\n' "${existing}" | sed 's/^-A /-D /' |
    while read -r rule; do iptables ${rule} || true; done
fi

for d in ${DNS}; do
  iptables -I DOCKER-USER -i "${BR}" -p udp --dport 53 -d "${d}" -j ACCEPT
  iptables -I DOCKER-USER -i "${BR}" -p tcp --dport 53 -d "${d}" -j ACCEPT
done
iptables -I DOCKER-USER -i "${BR}" -p tcp --dport 443 -d "${API}" -j ACCEPT
iptables -A DOCKER-USER -i "${BR}" -j DROP

[Unit]
Description=Egress allowlist for the agent bridge
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/agent-egress.sh

[Install]
WantedBy=multi-user.target

sudo chmod +x /usr/local/sbin/agent-egress.sh
sudo systemctl daemon-reload && sudo systemctl enable --now agent-egress.service

Prove it is idempotent by restarting it and comparing the chain to itself:

sudo systemctl restart agent-egress.service && sudo iptables -S DOCKER-USER

Six rules, twice, no duplicates.

Two things this buys quietly. getent runs on every start, so the API address is re-resolved at boot — not continuous, but the cheap majority of the moving-target problem with no extra daemon. And the delete loop strips only rules matching -i br-agent, so the unit never touches anything else in the chain.

If you switch Docker to the nftables backend​

Docker 29 ships an nftables firewall backend alongside iptables. It is not the default and this post did not run it — the switch rebuilds every rule on the host, and verifying the full behaviour needs a reboot. What follows is from Docker's documentation, flagged as such.

Everything above stops applying, because:

In Docker's nftables implementation, there is no DOCKER-USER chain.

You create your own table instead, and the priority does the work DOCKER-USER used to:

If your rules need to run before Docker's rules, give the base chains a lower priority number than Docker's chain.

And a rule that would be final under iptables is not:

In nftables, an "accept" rule is not final. It terminates processing for its base chain, but the accepted packet will still be processed by other base chains, which may drop it.

Overriding Docker's drop then requires --bridge-accept-fwmark.

The trap worth knowing about is the transition, and it is quiet in both directions:

When starting the daemon with nftables after running with iptables, Docker will not remove the jump from the FORWARD chain to DOCKER-USER. So, rules created in DOCKER-USER will continue to run until the jump is removed or the host is rebooted. When starting with nftables, the daemon will not add the jump. So, unless there is an existing jump, rules in DOCKER-USER will be ignored.

Switch backends and your policy keeps working — until the next reboot, when it silently stops. iptables -S DOCKER-USER will still list every rule.

Troubleshooting — the errors this run actually produced​

The systemd unit fails on its first run and works on the second​

status=1/FAILURE with nothing else in the journal. The cleanup line is the cause: on an empty chain, grep finds no matching rules and exits 1, which set -euo pipefail turns into an abort before any rule is added. The idempotency guard breaks the one run that needs no idempotency. See the || true in step 5.

Everything times out, including things you allowed​

Check whether the failure says "Resolving" or "Connection". If it says Resolving, your DNS rule is the problem and the destination rule was never reached. Containers frequently do not use the resolver you expect — read /etc/resolv.conf inside the container, not on the host.

The rules are listed but nothing is filtered​

Check that the bridge is up: ip -br link show br-agent. A network with no attached container has a DOWN interface, and a rule matching -i on it matches nothing.

Everything takes exactly eight seconds to fail​

That is DROP behaving correctly. Use REJECT while iterating if you would rather fail fast.

The measurement shows tens of milliseconds of overhead​

You are measuring DNS. Use %{time_connect} minus %{time_namelookup}, and be suspicious of any result where the minimums match but the medians don't.

What this did and didn't buy you​

Done: a container that can reach exactly one destination and a resolver; an exfiltration attempt that fails after a successful DNS lookup rather than before it; a policy that reapplies itself after a reboot and re-resolves its target when it does; and a measured cost indistinguishable from zero.

Not done:

• DNS is still an exfiltration channel. We allowed queries to 8.8.8.8, and data can be smuggled inside them. Closing that means a resolver you control and a rule pointing only at it — which is a different post.
• The allowlist is an IP, and IP addresses move. The unit re-resolves at boot, which is not the same as following a CDN. A forward proxy filtering on TLS SNI is the honest answer for a destination that rotates; it is also considerably more machinery.
• This protects containers. The agents in agent orchestration part 3 and part 4 run as host processes, where these rules do not apply. Moving them into a container is the prerequisite, and is worth doing on its own merits.
• The host is unrestricted, and so is every other container on this box. Five other bridges still have an unconditional ACCEPT in DOCKER-FORWARD.
• Nothing here stops the injection. It stops the payload leaving. Those are different jobs.

What's next​

The obvious sequel is the resolver: run one you control, point the container at it, allow port 53 only to that address, and the DNS channel closes along with the dependency on Google. It also sets up domain-based allowlisting properly, which is the piece of the moving-target problem this post left open.

Further reading​

• Docker — Packet filtering and firewalls
• Docker — Docker with iptables
• Docker — Docker with nftables
• Docker — Bridge network driver options
• OWASP — LLM06:2025 Excessive Agency
• NVD — CVE-2025-32711 (EchoLeak)
• NVD — CVE-2024-8309 (LangChain)

Part 5 gave an agent its own identity at the gateway: a token issued by authentik over client credentials, carrying a scope, expiring in five minutes. It ended by naming what it had not covered — the MCP tools server from agent orchestration part 4 still trusts anything that can reach its port, issue_refund included.

That post was explicit about the limit of what it had built:

The MCP server still trusts everyone. Scoping happens in the client. Anything that can reach 127.0.0.1:8770 can call issue_refund directly, agent or not.

Splitting the toolbox per role stopped an agent from reaching a tool it shouldn't. It did nothing about a curl. This part closes that.

One client or two​

The obvious move is to reuse Part 5's agent-gateway client and add a second scope to it. It already exists, the agent already holds its credentials, and it would work.

Don't. If one credential opens both the gateway and the tools server, then a leaked gateway key also issues refunds, and the blast radius of losing it doubles. The whole thread running through Part 4's groups and Part 5's grant types is that a credential should do one job.

So the tools server gets its own client, its own scope, and — because in authentik every provider is its own issuer — its own issuer. That last part turns out to matter more than expected, and we get to it in Step 4.

Prerequisites​

• The authentik from Part 3, with the agent-gateway client from Part 5
• The MCP server and virtualenv from agent orchestration part 3
• fastmcp 4.0.0 or newer — the client-credentials helper discussed in Step 5 does not exist before it

Step 1 — A second client, in three parts​

This is Part 5's procedure with different names, so it is deliberately terse here. If any of it is unfamiliar, Part 5 walks through the same three screens with figures.

The provider. Applications → Providers → New Provider → OAuth2/OpenID → Next. Name agent-tools, Client Type Confidential, Redirect URIs empty, and in Grant Types uncheck everything except Client credentials.

The scope. Customization → Property Mappings → New Property Mapping → Scope Mapping. Mapping Name mcp-invoke, Scope name mcp:invoke, expression return {}. The two name fields are different things: the first is authentik's label, the second is the string that lands in the token.

The application. Applications → Applications → New Application ▾ → with Existing Provider…, named Agent Tools, slug agent-tools typed by hand, provider agent-tools, hidden from the dashboard. Then Edit the provider → Advanced protocol settings → Scopes, and move mcp-invoke into Selected.

Verify the grant types from the database rather than the form, which is Part 4's lesson:

cd ~/authentik && docker compose exec server ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
for p in OAuth2Provider.objects.all():
    print(f'{p.name}: {p.grant_types}')" 2>/dev/null | tail -5

Provider for Langfuse: ['authorization_code', 'implicit', 'urn:ietf:params:oauth:grant-type:device_code']
Provider for LiteLLM: ['authorization_code', 'implicit', 'hybrid', 'refresh_token', 'client_credentials', 'password', 'urn:ietf:params:oauth:grant-type:device_code']
agent-gateway: ['client_credentials']
agent-tools: ['client_credentials']

Two clients trimmed to one grant each, and — visible in the same output — the LiteLLM provider from Part 4 still carrying password and five others nobody chose. Part 5 said to go back and fix those. This is what not doing it looks like.

Step 2 — A token, and why it is a different token​

Read the credentials out and keep them beside the gateway's, in their own directory:

cd ~/authentik
read -r TID TSEC < <(docker compose exec -T server ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider as P
p = P.objects.get(name='agent-tools')
print(p.client_id, p.client_secret)" 2>/dev/null | tail -1)

mkdir -p ~/mcp-auth && cd ~/mcp-auth
umask 077
cat > .env <<EOF
AK_TOKEN_URL=http://10.80.4.212:9100/application/o/token/
AK_CLIENT_ID=$TID
AK_CLIENT_SECRET=$TSEC
EOF
cp ~/agent-auth/token.sh ~/mcp-auth/token.sh

The helper is Part 5's, unchanged — it takes a scope and returns an access token.

~/mcp-auth/token.sh mcp:invoke | jq -R 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson'

{
  "iss": "http://10.80.4.212:9100/application/o/agent-tools/",
  "sub": "545a532b77a0f877339a5e4312970069deeb9ded604494c510debdfdce9c970d",
  "aud": "xNxmOEyPMPVThmxgLsyvTCHa3qvnpVjQSOY0bZck",
  "exp": 1789050030,
  "iat": 1789049730,
  "scope": "mcp:invoke"
}

Three fields differ from the gateway's token and all three are load-bearing. The issuer ends /agent-tools/ rather than /agent-gateway/. The audience is a different client ID. The scope is mcp:invoke. exp minus iat is 300 seconds, same as Part 5 — remember that number, it comes back in Step 5.

Step 3 — The verifier​

FastMCP validates JWTs with JWTVerifier, which takes the four things authentik just told us. Back up the server first, since it is the artefact two published parts are built on:

cp ~/mcp-tools/office_tools.py ~/mcp-tools/office_tools.py.bak

Its configuration goes in a file rather than the source:

cd ~/mcp-tools
umask 077
cat > .env.mcp <<'EOF'
MCP_JWKS_URI=http://10.80.4.212:9100/application/o/agent-tools/jwks/
MCP_ISSUER=http://10.80.4.212:9100/application/o/agent-tools/
MCP_AUDIENCE=<the agent-tools client id>
EOF

Then two changes to office_tools.py — an import, and a verifier where the bare constructor was:

from fastmcp import FastMCP, Context
from fastmcp.server.auth.providers.jwt import JWTVerifier

verifier = JWTVerifier(
    jwks_uri=os.environ["MCP_JWKS_URI"],
    issuer=os.environ["MCP_ISSUER"],
    audience=os.environ["MCP_AUDIENCE"],
    required_scopes=["mcp:invoke"],
)

mcp = FastMCP("office-tools", auth=verifier)

That is the entire server-side change. JWTVerifier fetches the JWKS, matches each token's kid to a key, and checks signature, issuer, audience, expiry and scope before any tool runs.

Restart with the new environment loaded:

pkill -f office_tools.py
cd ~/mcp-tools && set -a && source .env.mcp && set +a && \
  nohup ./.venv/bin/python -u office_tools.py > server.log 2>&1 &
sleep 4 && tail -8 server.log

Read that startup log carefully, because of what is missing from it. It reports the transport, the mode and the port, and says nothing whatsoever about authentication. A server with a verifier attached and a server without one log the same lines. There is no "auth enabled" message to look for, so the only way to know the verifier took is to make a request — which is the next step.

Step 4 — Three requests​

The same shape as Part 5's three status codes, because it reads well and because the middle one is the interesting one:

TOOLS=$(~/mcp-auth/token.sh mcp:invoke)
GW=$(~/agent-auth/token.sh gateway:invoke)

req() {
  curl -sS -o /dev/null -w "$1  HTTP %{http_code}\n" -X POST http://127.0.0.1:8770/mcp \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json, text/event-stream' \
    ${2:+-H "Authorization: Bearer $2"} \
    -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
}

req "no token     "
req "gateway token" "$GW"
req "tools token  " "$TOOLS"

no token       HTTP 401
gateway token  HTTP 401
tools token    HTTP 200

The first line is the gap agent orchestration part 4 admitted to, now closed. Before this change that request returned the full five-tool catalogue to anybody who asked — and with the right token, it still does:

curl -sS -X POST http://127.0.0.1:8770/mcp \
  -H "Authorization: Bearer $TOOLS" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \
  | tr -d '\r' | sed -n 's/^data: //p' | jq '.result.tools | length, (.[].name)'

Same five tools as before. Authentication changed who may ask, not what the server does.

The second is the one worth pausing on. That is a valid, unexpired, correctly signed authentik token that works perfectly well against the gateway, and the tools server refuses it. Ask the server why — sending the request again first, so the answer is at the end of the log rather than buried under whatever has hit the server since:

GW=$(~/agent-auth/token.sh gateway:invoke)

curl -sS -o /dev/null -w 'gateway token: HTTP %{http_code}\n' -X POST http://127.0.0.1:8770/mcp \
  -H "Authorization: Bearer $GW" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

tail -12 ~/mcp-tools/server.log

WARNING  Bearer token rejected for client aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ
         : issuer mismatch (got 'http://10.80.4.212:9100/application/o/agent-gateway/',
                       expected 'http://10.80.4.212:9100/application/o/agent-tools/')
INFO     Auth error returned: invalid_token (status=401)

It failed on iss, not on scope. That is the separate provider paying off. In authentik each provider is its own issuer, so a token from the wrong client is rejected at the first gate, before audience or scope are looked at. Had we taken the easy road and added a second scope to the gateway's client, the same request would have got several checks further before required_scopes turned it away. Same outcome, more surface.

Two other things in that log. It names the offending client by ID, and prints both the issuer it got and the one it wanted — so debugging is a tail, not a guess. And the caller receives only invalid_token, which is correct: telling a client which issuer you expected is telling an attacker what to forge.

Step 5 — The client half, and a helper that cannot help​

The tools server is now protected, which means every script from part 3 and part 4 is broken. They connect with no token.

FastMCP documents exactly the right thing for this: ClientCredentialsOAuthProvider, which runs the client-credentials grant itself and — the part that matters given our 300-second tokens — "when the token expires it is re-acquired automatically on the next request."

It does not work here. Try it and you get:

mcp.client.auth.exceptions.OAuthTokenError: Token exchange failed (404): Not Found

The server log explains what actually happened:

GET  /.well-known/oauth-authorization-server      404
POST /token                                        404
GET  /.well-known/oauth-protected-resource         404
GET  /.well-known/oauth-protected-resource/mcp     404
GET  /.well-known/openid-configuration             404

The provider takes the MCP server's URL, not a token endpoint, because "the token endpoint is discovered from the server's OAuth metadata." Our server publishes none — and FastMCP's own documentation says so, on a different page: "TokenVerifier focuses exclusively on token validation without providing OAuth discovery metadata."

So the two pages are consistent and the combination is a documented dead end. You just have to read both to find that out, and the failure tells you none of it: four 404s you only see if you are watching the server, a fallback that POSTs /token at the resource server, and an error that never mentions discovery.

The path the docs point to instead is the client obtaining its token separately. BearerAuth takes a token you already have and attaches it to every request — worth proving standalone before wiring it into four scripts:

That works, so it can be shared. One helper, used by every script in the series:

"""One authenticated MCP client, shared by every agent in the series."""
import os
import subprocess

from fastmcp import Client
from fastmcp.client.auth import BearerAuth

MCP_URL = "http://127.0.0.1:8770/mcp"
TOKEN_SH = os.path.expanduser("~/mcp-auth/token.sh")


def fetch_token(scope: str = "mcp:invoke") -> str:
    """Mint a short-lived token from authentik via the client credentials grant."""
    out = subprocess.run([TOKEN_SH, scope], capture_output=True, text=True, check=True)
    return out.stdout.strip()


def authed_client() -> Client:
    return Client(MCP_URL, auth=BearerAuth(token=fetch_token()))

Each agent then changes by two lines — an import, and what it hands MCPAdapter:

from mcp_auth import authed_client

async with MCPAdapter(authed_client()) as adapter:

MCPAdapter accepts a configured Client as readily as a URL string, which part 3 used for caching without needing it for anything else. Here it is what lets the token in.

BearerAuth is the explicit form. FastMCP also accepts the token as a bare string — Client(url, auth=token) — and adds the scheme itself; its documentation is specific that you "do not include the Bearer prefix" if you do. The class is worth the extra import here only because it makes the intent obvious at the call site.

Step 6 — Prove a tool actually runs​

Discovery is not execution. A token that lists tools might still fail at the call, so check the thing that touches the database:

TOOLS=$(~/mcp-auth/token.sh mcp:invoke)

curl -sS -X POST http://127.0.0.1:8770/mcp \
  -H "Authorization: Bearer $TOOLS" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"find_customer","arguments":{"query":"Maria"}}}' \
  | tr -d '\r' | sed -n 's/^data: //p' | jq -c '.result.content[0].text // .error'

"[{\"id\":1,\"name\":\"Maria Alvarez\",\"email\":\"maria@example.com\"}]"

Drop the Authorization header and the same call returns 401. A real row with a token, nothing without — that is authentication reaching execution, not just the catalogue.

Troubleshooting — the errors this run actually produced​

Token exchange failed (404): Not Found​

ClientCredentialsOAuthProvider against a JWTVerifier server. Discovery found nothing and it fell back to guessing. See Step 5; use BearerAuth.

2 validation errors ... Missing required argument​

The token was fine — this comes from inside the tool. params.name is the tool's name, params.arguments are its parameters, and both nest a field called name. find_customer takes query.

The log says only invalid_token​

You grepped it. See the warning in Step 4.

Every script from parts 3 and 4 suddenly fails​

Expected. They connect without a token. Step 5.

What this did and didn't buy you​

Done: an MCP server that authenticates every caller, a credential that is not the gateway's, a refusal that happens at the issuer rather than deep in the scope check, and a log that names which client was turned away and why.

Not done:

• No per-tool authorisation, and this is less than agent orchestration part 4 promised. That post closed by saying the next step was to "authenticate the tool call, so the server knows which agent is asking and refuses issue_refund to anything that isn't billing." Half of that is now true: the call is authenticated. The other half is not. required_scopes is a property of the server, not of a tool, so every holder of mcp:invoke can call issue_refund exactly as easily as find_customer, and the server still cannot tell the scheduler from the billing agent. Splitting tools per agent, as part 4 did, remains a client-side arrangement.
• Tokens still expire mid-run, as described in Step 5.
• Still plain HTTP. A bearer token in the clear is a credential in the clear.
• The gateway's own provider is still over-permissioned, visible in Step 1's output and unfixed since Part 5 pointed at it.

What's next​

Per-tool authorisation is the obvious gap and it does not have an obvious answer. required_scopes gates the server; gating issue_refund differently from find_customer means either a second server with its own client, or authorisation inside each tool body reading the verified claims. The first is more boxes and a clean boundary. The second is a check on every tool you ever write, which is precisely the failure mode part 4 warned about for policy-in-the-tool.

Further reading​

• FastMCP — token verification
• FastMCP — machine-to-machine client authentication
• FastMCP — bearer token authentication
• authentik — client credentials
• RFC 9728 — OAuth 2.0 Protected Resource Metadata

Part 4 put group access control on the gateway's admin UI and then, at the end, called the API from a shell with no account, no session and no group. It answered normally. The conclusion was that SSO protects a control plane and the data plane authenticates machine callers with keys — which it has to, because an agent running at 3am cannot complete a browser login.

That was true and it was also a stopping point rather than an answer. "Machine callers use keys" leaves you with a credential that never expires, that no identity provider knows about, and that survives the person who created it. Part 4 said so plainly: removing someone from a group does not revoke their keys, a leaked key is unaffected by identity entirely, and keys outlive people.

This part gives the machine an identity instead of a key.

authentik issues the agent a token over the client credentials grant — no browser, no consent screen, no human. The token is signed, carries a scope, and expires in five minutes. The gateway verifies it locally against authentik's public keys and refuses anything without the right scope. At the end, three status codes show the boundary holding.

Two grants, two different questions​

Parts 3 and 4 used the authorization code grant throughout. A person clicks a button, gets redirected to authentik, types a password, sees a consent screen, and is redirected back with a code the application exchanges for a token. Every step of that assumes a browser and a human attached to it.

The client credentials grant answers a different question. There is no user. The client is the principal, and it proves that with an ID and a secret, in one request, with no redirect anywhere.

The property worth noticing: the gateway never asks authentik whether a token is good. It fetches authentik's public keys once, caches them, and checks the signature itself. Identity scales without the identity provider becoming a bottleneck — or a single point of failure on every request.

Prerequisites​

• A working authentik, ideally the one from Part 3
• A running LiteLLM gateway from Part 1 of the gateway series
• Its LITELLM_MASTER_KEY, and shell access to the box

Step 1 — A client with exactly one grant​

Applications → Providers → New Provider → OAuth2/OpenID Provider → Next.

• Provider Name — agent-gateway
• Authorization Flow — leave default-provider-authorization-explicit-consent. It is required, and it is inert here: it governs the consent screen a human sees, and no human will ever use this client. You have to pick one anyway.
• Client Type — Confidential. authentik's own description is the reason: "Confidential clients are capable of maintaining the confidentiality of their credentials such as client secrets." Client credentials has nothing but that secret to prove identity, so a public client cannot use this grant at all.
• Redirect URIs — leave empty. Nothing is ever redirected anywhere.

Then scroll to Grant Types, and look at what a new provider ships with.

Seven of eight, enabled by default. Two of them — Implicit and Password — did not make it into OAuth 2.1, which states plainly that "some features available in OAuth 2.0, such as the Implicit or Resource Owner Credentials grant types, are not specified in OAuth 2.1" (§10.1). Password in particular has the client handle a user's actual password, which is why it fell out of favour. Nothing asked whether you wanted them and nothing warns you that they are on.

Uncheck everything except Client credentials.

This is the same least-privilege reflex Part 4 applied to group membership, one layer down. A client that only ever uses one grant should only be permitted one grant, so that a misconfiguration elsewhere cannot turn it into a password-accepting endpoint.

cd ~/authentik && docker compose exec server ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
for p in OAuth2Provider.objects.all():
    print(f'{p.name}: {p.grant_types}')" 2>/dev/null | tail -5

Provider for Langfuse: ['authorization_code', 'implicit', 'urn:ietf:params:oauth:grant-type:device_code']
Provider for LiteLLM: ['authorization_code', 'implicit', 'hybrid', 'refresh_token', 'client_credentials', 'password', 'urn:ietf:params:oauth:grant-type:device_code']
agent-gateway: ['client_credentials']

Step 2 — The application it belongs to​

Read back the credentials and try to get a token. It will not work yet, and the reason is the most useful thing in this post.

mkdir -p ~/agent-auth
cd ~/agent-auth
umask 077
cat > .env <<EOF
AK_TOKEN_URL=http://10.80.4.212:9100/application/o/token/
AK_CLIENT_ID=<client id from the provider page>
AK_CLIENT_SECRET=<client secret from the provider page>
EOF

umask 077 creates the file with 600 permissions rather than chmod-ing it afterwards, so there is no window where the secret is world-readable.

A small helper, because you will fetch tokens constantly:

cat > ~/agent-auth/token.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.env"

curl -sS -X POST "$AK_TOKEN_URL" \
  -d grant_type=client_credentials \
  -d client_id="$AK_CLIENT_ID" \
  -d client_secret="$AK_CLIENT_SECRET" \
  -d scope="${1:-}" \
  | jq -r .access_token
EOF
chmod +x ~/agent-auth/token.sh

Ask for a token:

cd ~/agent-auth
source .env
curl -sS -X POST "$AK_TOKEN_URL" \
  -d grant_type=client_credentials \
  -d client_id="$AK_CLIENT_ID" \
  -d client_secret="$AK_CLIENT_SECRET" | jq

{
  "error": "invalid_grant",
  "error_description": "The provided authorization grant or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client",
  "request_id": "757c397b97a745259e288b6d4ff1553c"
}

Every clause in that description is wrong for our situation. Nothing is expired, nothing is revoked, no redirection URI was used, and the client is the one the credentials belong to. The server log records the 400 and no reason. authentik's Events show nothing useful either.

The actual cause is visible in the Providers list, which flags it plainly:

A provider with no application cannot issue a token. authentik authorizes against the application, so a provider on its own has nothing to authorize against — and says invalid_grant instead of saying that.

This is not in authentik's client credentials documentation, which documents five ways to authenticate — three static-credential variants and two JWT ones — and says nothing about needing an application.

Pair one. Note the route: the New Application wizard always creates a new provider, so it cannot adopt the one we just built.

Its step list gives it away — Choose a Provider is step 2, and every option there builds a new one. The caret beside the button is where the other path lives.

Applications → Applications → New Application ▾ → with Existing Provider…

• Name — Agent Gateway
• Slug — agent-gateway, typed explicitly. Part 4's finding was that authentik derives slugs from names and turned LiteLLM into lite-llm; the slug lands in the issuer URL, so it is worth setting by hand every time.
• Provider — agent-gateway
• Hidden from Application Dashboard — check it. No person should ever see a tile for a machine client.

Retry the token request and it succeeds. Nothing changed except the pairing.

Step 3 — A scope that means something​

The token you now get carries "scope": "". It proves who the caller is and grants no authority at all. A gateway receiving it knows the client is agent-gateway and nothing more.

Customization → Property Mappings → New Property Mapping → Scope Mapping.

• Mapping Name — gateway-invoke, authentik's label for the object
• Scope name — gateway:invoke, the string clients request and that appears in the token
• Expression — return {}

The two name fields are different things and the form does not make that obvious. The expression is Python whose return value is merged into the token's claims; an empty dict is right here, because we want the scope named in the token, not extra data carried inside it.

Attach it: Applications → Providers → agent-gateway → Edit → Advanced protocol settings → Scopes, and move gateway-invoke from Available to Selected.

Read the help text under that pane, because it is the whole mechanic:

"Select which scopes can be used by the client. The client still has to specify the scope to access the data."

Attaching a scope makes it available. The client still has to ask. That is why the first token came back empty even though three scopes were already attached — nothing had requested them.

Now ask for it:

TOKEN=$(~/agent-auth/token.sh gateway:invoke)
jq -R 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson' <<< "$TOKEN"

{
  "iss": "http://10.80.4.212:9100/application/o/agent-gateway/",
  "sub": "2451c6a1bab3085590263f119e650448beb013336146bcfee8241dd5832c200b",
  "aud": "aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ",
  "exp": 1788901322,
  "iat": 1788901022,
  "acr": "goauthentik.io/providers/oauth2/default",
  "jti": "NRDTukijwLJbW7gpPtn9zUZwzTj6d2kBSZCuFYvX",
  "azp": "aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ",
  "uid": "neugHWoWQOQzADgbV1vKwzew2COQ1RgCXfyywgqa",
  "scope": "gateway:invoke"
}

exp minus iat is 300 seconds. That is the entire argument for this over a static key: a token pasted into a chat window, a log file or a screenshot is worthless five minutes later.

There is no email or preferred_username in there, because there is no user. sub is a service account authentik created for you, named ak-agent-gateway-client_credentials — it appears in Directory → Users without ever being asked for.

Step 4 — Teaching the gateway to verify​

LiteLLM documents JWT authentication with a litellm_jwtauth block, enforce_scope_based_access and scope_mappings. Configure it on a stock install and nothing happens, because the same page says:

"JWT-based Auth requires a LiteLLM Enterprise license."

Check before you write config:

docker exec llm-gateway python -c "
from litellm.proxy.proxy_server import premium_user
print('premium_user =', premium_user)"

premium_user = False

False means that whole documented path is closed to you. The open-source alternative is custom_auth, which hands every request to a function you write — and for this job it is about twenty-five lines.

import os

import jwt
from fastapi import Request
from litellm.proxy._types import UserAPIKeyAuth

JWKS_URL = os.environ["JWT_PUBLIC_KEY_URL"]
ISSUER = os.environ["JWT_ISSUER"]
AUDIENCE = os.environ["JWT_AUDIENCE"]
REQUIRED_SCOPE = "gateway:invoke"
MASTER_KEY = os.environ["LITELLM_MASTER_KEY"]

_jwks = jwt.PyJWKClient(JWKS_URL)


async def user_api_key_auth(request: Request, api_key: str) -> UserAPIKeyAuth:
    token = api_key.removeprefix("Bearer ").strip()

    # Not a JWT: fall back to the master key so existing tooling keeps working.
    if token.count(".") != 2:
        if token == MASTER_KEY:
            return UserAPIKeyAuth(api_key=token)
        raise Exception("not a JWT and not the master key")

    signing_key = _jwks.get_signing_key_from_jwt(token)
    claims = jwt.decode(
        token,
        signing_key.key,
        algorithms=["RS256"],
        issuer=ISSUER,
        audience=AUDIENCE,
    )

    if REQUIRED_SCOPE not in claims.get("scope", "").split():
        raise Exception(f"token is missing scope {REQUIRED_SCOPE}")

    return UserAPIKeyAuth(api_key=token, user_id=claims["sub"])

PyJWKClient does the part that looks hard. It fetches the JWKS, matches the token's kid header to the right key, and caches it — so after the first request, verification is local arithmetic. jwt.decode checks the signature, the issuer, the audience and the expiry in one call, and raises if any of them is wrong.

Three environment variables, appended to ~/llm-gateway/.env:

JWT_PUBLIC_KEY_URL=http://10.80.4.212:9100/application/o/agent-gateway/jwks/
JWT_ISSUER=http://10.80.4.212:9100/application/o/agent-gateway/
JWT_AUDIENCE=aObZ8KvKsKVdQ4MuSsTINp0f65Puz5fzXUUBC6iQ

JWT_ISSUER and JWT_AUDIENCE must match the iss and aud you saw in the decoded token exactly, or every token is refused.

Mount the module the same way the gateway already mounts scrubber.py, in docker-compose.yml:

      - ./jwt_auth.py:/app/jwt_auth.py:ro

And in config.yaml, under general_settings:

  custom_auth: jwt_auth.user_api_key_auth
  custom_auth_run_common_checks: true

That second line matters more than it looks. Without it, LiteLLM warns at startup:

"custom_auth is configured but 'custom_auth_run_common_checks' is not set. Problem: budgets, model-access allowlists, and per-model rate limits configured on your DB team/project records will NOT be enforced for custom-auth requests."

Read that carefully. Adding token authentication would, by default, have exempted every token holder from the controls you already configured. You would have made the gateway more secure in one dimension and quietly weaker in another, and the only notice is one warning line at startup.

Step 5 — The boundary, in three status codes​

TOKEN=$(~/agent-auth/token.sh gateway:invoke)
NOSCOPE=$(~/agent-auth/token.sh)

curl -sS -o /dev/null -w 'no token      HTTP %{http_code}\n' http://127.0.0.1:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen-mid","messages":[{"role":"user","content":"Say OK."}]}'

curl -sS -o /dev/null -w 'wrong scope   HTTP %{http_code}\n' http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $NOSCOPE" -H 'Content-Type: application/json' \
  -d '{"model":"qwen-mid","messages":[{"role":"user","content":"Say OK."}]}'

curl -sS -o /dev/null -w 'valid token   HTTP %{http_code}\n' http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"model":"qwen-mid","messages":[{"role":"user","content":"Say OK."}]}'

no token      HTTP 401
wrong scope   HTTP 401
valid token   HTTP 429

The first two never reach a model. The second is the one that matters: a valid, correctly signed, unexpired token from the right issuer, refused because it did not carry gateway:invoke. Identity alone is not authority.

The third is the interesting one, and it is not a 200. It cleared authentication and scope, and was then answered by a completely different part of the gateway — the budget and routing layer the gateway series configures, which on this box had a cap already reached. That is the point of the shape rather than a flaw in it: a 401 means we do not know who you are, or you may not do this, and anything else means the request got past identity entirely and is now somebody else's decision. Authentication's job ends at the first two lines.

Confirm the refusal came from your check rather than something incidental:

docker logs llm-gateway --since 10m 2>&1 | grep -iE 'missing scope|401 Unauthorized' | tail -5

user_api_key_auth(): Exception occured - token is missing scope gateway:invoke
Exception: token is missing scope gateway:invoke
INFO:     "POST /v1/chat/completions HTTP/1.1" 401 Unauthorized

Note where that reason lives. The caller receives a bare 401 with no explanation, which is correct — telling a client "your token is valid but lacks scope X" tells an attacker exactly what to go and get. It does mean that debugging this always means reading the gateway log, and never the response body.

Troubleshooting — the errors this run actually produced​

invalid_client​

The credentials did not authenticate at all. Before blaming authentik, check what actually landed in your .env:

awk -F= '{printf "%-18s %3d chars\n", $1, length($2)}' ~/agent-auth/.env

AK_TOKEN_URL        48 chars
AK_CLIENT_ID        40 chars
AK_CLIENT_SECRET   128 chars

The client ID is 40 characters and the secret is 128. A zero-length value means the file is the problem, not the provider.

invalid_grant, with everything apparently correct​

The provider is not paired to an application. See Step 2 — the Providers list flags it, the error text does not.

AK_TOKEN_URL: unbound variable, and a corrupted secret​

echo 'X=1' >> .env on a file whose last line has no trailing newline appends to that line instead of creating a new one:

AK_CLIENT_SECRET=p94oDU...dgJcAK_TOKEN_URL=http://10.80.4.212:9100/application/o/token/

Two variables destroyed in one command, silently — the secret is now wrong and the new variable does not exist. Check with cat -A, which marks line ends with $, and prefer writing the whole file with a single heredoc over appending to it.

The config change had no effect​

The container was never restarted. See the warning in Step 4.

401 with no reason in the response​

By design. The reason is in docker logs llm-gateway.

What this did and didn't buy you​

Done: a machine identity issued by your identity provider rather than minted by hand, a credential that expires in five minutes, a scope that has to be requested and is checked on every call, revocation that happens in one place, and a caller the gateway can name in its logs as something other than a key prefix.

Not done:

• The MCP server is still open. Everything here protects the gateway. The tools server from agent orchestration Part 4 still trusts anything that can reach its port, issue_refund included. Same problem, one layer up, and the subject of the next part.
• One scope, one client. A real deployment has several agents with different scopes, and scope_mappings between scopes and models is where that goes.
• The master key still works, deliberately, as the fallback in the handler. It remains the break-glass credential Part 4 described, and it still bypasses everything.
• Still plain HTTP. The token crosses the network in the clear, and a token in the clear is a key in the clear until it expires.
• The service account is invisible. authentik created ak-agent-gateway-client_credentials without asking, and nothing in the token says it is a machine rather than a person.

What's next​

The MCP server. FastMCP ships a JWTVerifier that takes a jwks_uri, an issuer, an audience and required_scopes — the same four things configured here — so the tools server can demand a scoped token before it will list a tool, let alone run one. That closes the gap agent orchestration Part 4 left open, where scoping happened in the client and the server trusted whoever reached the port.

Further reading​

• authentik — client credentials
• authentik — provider property mappings, and scope mappings with OAuth2
• LiteLLM — custom auth
• LiteLLM — OIDC JWT-based auth, the Enterprise path
• PyJWT — PyJWKClient
• RFC 6749 §4.4 — the client credentials grant

Part 3 gave an agent five tools over MCP — a calendar, a customer list, invoices, and a refund — and put a human in front of the refund. It ended by admitting the obvious: one agent held all five. Nothing stopped the model reaching for issue_refund when it had been asked to book an appointment. The human pause was the only thing in the way, and a pause only fires if the model calls the tool at all.

This part removes the tool instead. The scheduler gets three tools and the billing agent gets three, sharing one. Ask the scheduler to refund an invoice and it cannot — not because it was told not to, not because a policy blocked it, but because issue_refund was never in the list it was given.

Then Part 2's supervisor goes back on top, and the interesting property falls out: routing decides who works, scoping decides what is possible. A misrouted refund still cannot refund.

What "scoping" actually means here​

There is no new mechanism in this post. create_agent takes a list of tools; we hand it a shorter list. That is the entire technique, and it is worth being precise about why it is stronger than the alternatives:

• A system prompt telling the model not to issue refunds is a request. It survives exactly as long as the model cooperates.
• A policy check inside the tool is real, but it runs after the model decided to call it, and it has to be written into every tool you ever add.
• Not passing the tool removes the option from the model's context. There is nothing to refuse, because there is nothing to call.

The last one is the only one that doesn't depend on inference going well.

One server, two toolboxes​

The server offers all five to anyone who asks. What differs is which ones each agent is handed. find_customer goes to both, because looking a customer up is harmless.

The blocked arrow is the whole post. The scheduler cannot call issue_refund — not because a rule forbids it, but because the tool is not in the list it was given, so it never reaches the model's context at all.

Prerequisites​

• The MCP server, virtualenv and seeded database from Part 3
• The gateway key in ~/mcp-tools/.env, as Part 3 set up
• Python 3.10+

Part 3 ran the server in the foreground, which cost a terminal and died with the SSH session. Start it detached instead — one session is enough for everything below:

cd ~/mcp-tools && nohup ./.venv/bin/python office_tools.py > server.log 2>&1 & sleep 3; ss -tlnp | grep 8770

Step 1 — Two toolsets from one server​

The server still offers five tools. The roles name subsets of them:

import asyncio, os, sys
from langchain.agents import create_agent
from langchain.mcp import MCPAdapter
from langchain_openai import ChatOpenAI
from langgraph.checkpoint.memory import InMemorySaver

MCP_URL = "http://127.0.0.1:8770/mcp"

ROLES = {
    "scheduler": ("find_customer", "list_availability", "book_slot"),
    "billing":   ("find_customer", "open_invoices", "issue_refund"),
}

model = ChatOpenAI(
    base_url="http://127.0.0.1:4000/v1",
    api_key=os.environ["GATEWAY_KEY"],
    model="qwen-mid",
    temperature=0,
)

async def main():
    role, request = sys.argv[1], sys.argv[2]
    async with MCPAdapter(MCP_URL) as adapter:
        catalog = {t.name: t for t in await adapter.list_tools()}
        print(f"server offers : {sorted(catalog)}")

        allowed = [catalog[n] for n in ROLES[role]]
        print(f"{role} can see : {[t.name for t in allowed]}\n")

        agent = create_agent(model, allowed, checkpointer=InMemorySaver())
        cfg = {"configurable": {"thread_id": f"{role}-1"}}
        out = await agent.ainvoke({"messages": [{"role": "user", "content": request}]}, cfg)

        if "__interrupt__" in out:
            req = out["__interrupt__"][0].value["requests"][0]
            print(f"PAUSED — the server is asking a human:\n  {req['message']}")
            return

        print("--- answer ---")
        print(out["messages"][-1].content)

asyncio.run(main())