![\"A](\"https://development-wec.wiline.com/docs/img/news/litellm-cover.webp\")
TypeSafe shipped its first model on 15 September,\nand the interesting thing about Jev is what it refuses to do. It doesn't write you a\nparagraph. You give it a request and it hands back one structured value — a label, a\nclass, a decision — in, they say, 70 to 500 ms. Founder Diogo Almeida's framing is\nthe clearest line in the post: \"Think of Jev as a frontier-intelligence function\ncall: unstructured state in, typed probabilistic decisions out.\"
\nMost models are built to talk to people. Jev is built to be called by code — and the\nfirst job that shape fits is routing.
\n\nThe name tells the story twice. \"System One\" is Kahneman's term for fast, automatic\njudgement — the snap decision — against \"System Two,\" the slow deliberate reasoning\nwe reach for a big LLM to do. And \"Jev\" is for William Stanley Jevons, the\neconomist of the efficiency paradox: make something cheaper and people use far more\nof it. TypeSafe is betting cheap machine decisions get used everywhere.
\nA chat model generates tokens one at a time until a paragraph exists. Jev does the\nopposite — TypeSafe says it \"gives up string generation,\" samples all outputs in a\nsingle parallel query, and returns one value from a set you define in advance. Two\nproperties fall out, and both matter for routing:
\nSIMPLE, MEDIUM,\nCOMPLEX, REASONING), not prose you have to parse. TypeSafe claims it \"never\nmakes type errors\" — worth being precise here, because they are: that 0% is\n\"not empirical. Schema matching is guaranteed, thus we can confidently add 0%.\"\nIt's a property of constraining the output to a schema, not a measured result.Their headline numbers (\"40x-200x faster,\" output \"too cheap to meter,\" a workflow\neval at \"193.6x faster, 444.6x cheaper\") are vendor claims on their own evals, and\nto their credit they flag the bias — the evals were \"made by individuals on our\nmodel capabilities team.\" Treat them as claims until someone independent re-runs\nthem.
\nModel routing is a cost play. Not every request needs your biggest model, so you put\na cheap model and an expensive one behind one endpoint and something in front decides\nwhich answers. Easy turns take the cheap tier; hard ones take the expensive tier; the\nbill drops. That \"something in front\" is a classifier:
\n\nBuild that classifier the usual way — ask another LLM \"is this simple or complex?\"\n— and every request pays for an LLM call before it pays for the answer. That call\nis slow and costs tokens, on every request, whether it landed on the cheap tier or\nnot. The decision meant to save money is quietly spending it.
\nJev fits that slot. You don't ask it to answer — you ask which tier the request\nbelongs to, and the gateway routes on the label:
\nIn LiteLLM's Auto Router that is a config change, not new code — you name the tiers,\nmap each to a model, and set classifier_type: jev:
- model_name: jev-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: {{openai_small}}
MEDIUM: {{openai_large}}
COMPLEX: {{anthropic}}
REASONING: {{anthropic_large}}
classifier_type: jev
On 20 September LiteLLM's Moe Khalil published a benchmark\ntitled \"JEV Classifier: 5.43x as Fast as Haiku, 96% Lower Cost.\" He ran 80 cases\nthree times each — 240 classification calls — with Jev (jev-1.13.0) against Claude\nHaiku 4.5 as the \"classify with an LLM\" baseline:
The average hides the real finding, which is in the per-tier numbers. On SIMPLE\nboth were perfect (100%). But on MEDIUM Haiku collapsed to 38.33% where Jev\nheld 85%, and on REASONING Haiku managed 76.67% against Jev's 100%. In\nother words, the LLM classifier is fine at telling trivial from non-trivial, and\nunreliable exactly in the middle band where routing decisions actually save or cost\nyou money.
Two caveats, both LiteLLM's own, stated plainly: the expected tiers \"were authored\nwith the synthetic prompts, without independent review,\" and the benchmark measured\nthe routing decision, not the quality of the final answer — \"downstream answer\nquality was not measured.\" To their credit they published a frozen, hash-verified\nreproduction archive so the\nrun can be re-analysed. Read the result as \"Jev picked the tier fast, cheap, and the\nway they expected,\" not \"your answers get 96% cheaper end to end.\"
\nWe build this exact setup in the tutorial on\nrouting by complexity — a gateway in front\nof the WEC Inference API that scores each request and sends it to the right tier,\nQwen3.5:9B for the simple turns and Qwen3.5:122B for the hard ones. And we have shown\nhow a classifier gets it wrong on a bare \"yes\",\nsending work to the wrong model.
\nIn both, the classifier is the part nobody prices. Put an LLM in that slot and you\nadd two-thirds of a second and a token charge to every call before the real model is\neven chosen. A model like Jev is a bet that the decision in front of your models can\nbe near-free — so routing pays off instead of taxing itself.
\nIt is early, it is hosted, and the numbers still need someone independent to re-run\nthem. For a WEC stack that keeps inference in-house, \"hosted\" is the real question\nmark — you would be sending every request's routing decision to a third party. But\nthe shape is the part that travels: a model that decides in milliseconds, sitting in\nfront of the models that answer.
\nPfizer runs an AI gateway — the single service every internal tool talks to when it\nwants a model. During a routine upgrade check, its throughput fell by half. No\nerrors. No failed requests. Nothing wrong on any dashboard.
\nThey published what they found\ntogether with the LiteLLM team, and the cause is small enough to fit in a paragraph:\nsomeone wrote a setting that said don't encrypt this connection, and the system\nencrypted it anyway.
\n\nThe gateway keeps a cache — a fast side-store called Redis that it checks before\ndoing expensive work. The connection to that cache can be encrypted or not, and\nthat's a setting you write in a config file.
\nThe operator wrote the setting to off. Plain connection, no encryption.
\nThe code that read that setting checked whether the setting existed, rather than\nwhat it was set to. Writing off creates the setting. The setting now exists. So\nthe check passed, and the gateway opened an encrypted connection — to a cache that\nwasn't expecting one.
Here's the part that makes this dangerous rather than merely wrong. When a program\ntries to start an encrypted conversation with something that doesn't speak\nencryption, it doesn't get turned away. It says hello and waits for a reply that\nnever comes. Not refused — ignored.
\nSo the cache wasn't down. The cache was slow. And slow is much harder to see.
\nFor anyone who wants to look at it directly, this is\nget_redis_connection_pool\nas it stood in the affected version:
connection_class = async_redis.Connection
if \"ssl\" in redis_kwargs:
connection_class = async_redis.SSLConnection
redis_kwargs.pop(\"ssl\", None)
redis_kwargs[\"connection_class\"] = connection_class
\"ssl\" in redis_kwargs asks does this key exist. It does — you created it when you\nset it to false. The branch fires, and you get the encrypted connection you\nexplicitly turned off.
Leave the setting out entirely and everything works. Write it down and say no, and it\nbreaks. The careful configuration is the broken one.
\nEvery layer above the cache is built to tolerate a cache that isn't answering, and\nthat's correct design — if the cache is unavailable, you do the work the slow way and\ncarry on. Which is exactly what turns this into silence:
\n\n\nRedis timeout errors appeared in the proxy's internal logs, but the gateway still\nreturned HTTP 200 to every caller
\n
HTTP 200 means success. Every single request reported success, for the entire\nduration. The post is precise about where the damage landed:
\n\n\nRedis operations were stalling on TLS handshake timeouts in the async hot path,\ndegrading throughput without surfacing HTTP-level errors
\n
The reported numbers: throughput down from about 300 requests per second to about\n156 — roughly half — with median response time under load at 4,200 ms, measured\nagainst 750 simulated concurrent users over a sustained 60-second run.
\nThose measurements come from Pfizer's internal testing and are reported in the post\nrather than independently reproducible. The code is a different matter — you can open\nthe file at that version and read it yourself.
\nTheir own one-line summary is the best sentence in the writeup:
\n\n\nA throughput drop with a 0% HTTP error rate is the worst kind of regression to\ncatch after the fact
\n
I wanted to know whether this arrived with the upgrade that exposed it. It didn't. I\ncompared the relevant file between the version that performed fine and the version\nthat didn't — they're byte-for-byte identical. The post says the same thing:
\n\n\nThe underlying Redis bug existed in older versions too, but surfaced during\nPfizer's v1.89.2 upgrade validation under this workload
\n
That's the detail worth sitting with. This wasn't a bad release you could roll back.\nThe flaw had been sitting in the connection code across versions that benchmarked\nperfectly well. It needed a particular configuration and a particular load before it\ndid any visible damage — and then it cost half a production gateway's capacity\nwithout raising anything.
\nNot through monitoring, and not by hunting through recent code changes. By turning\nfeatures off and on until the number moved:
\n\n\nEnabled Redis caching. Median latency jumped to 4,200ms. That isolated it to the\nRedis connection path.
\n
A team with real production traffic, real observability and a direct line to the\nvendor found this by hand. There was no signal in the error rate, because there\nwere no errors. Nothing in the status codes, because they all said success. The only\nplace it showed up was response time under sustained load — which you see only if\nyou go looking on purpose.
\nThat matches what I ran into at a far smaller scale when I\nload tested a gateway to find the stall the median hides:\nin this kind of infrastructure, the failures that cost you most are the ones that\nnever raise an error. A blocking callback and a connection stuck waiting on a\nhandshake are completely different bugs with the same fingerprint — throughput\ncollapses, the slowest requests get much slower, and the error rate never moves.
\nI should be straight about the limits here: I don't run anything at Pfizer's scale,\nso I can't reproduce their measurements. What I can do is read the code, and the code\nis public.
\nOne line, in the current version:
\nif redis_kwargs.pop(\"ssl\", None):
redis_kwargs[\"connection_class\"] = async_redis.SSLConnection
This version reads the setting's value instead of merely noting that it exists. Off\nnow means off.
\nIt's tracked as \"Redis ssl handling — Presence check → value check\" under LIT-4307\nand PR #32590, shipped in v1.93.0.\nThe pull request title puts it plainly: honor ssl value instead of key presence when\nbuilding async connection pool.
\nTwo things, if you run anything with a cache behind it.
\nCheck what your system actually did, not what you told it to do. Writing a\nsetting and having that setting take effect are different events, and this is a clean\nexample of the gap. The connection Pfizer got was the opposite of the one they asked\nfor, and nothing anywhere said so.
\nIf your testing measures averages, it cannot see this. Neither can alerting built\non error rates. Sustained load, slowest-request timings, and throughput compared\nagainst a known baseline are what turn a silent stall into something visible.
\nThe dashboards stayed green through the whole thing. That's not a gap you close by\nadding another alert on errors — there weren't any to alert on.
", "url": "https://development-wec.wiline.com/docs/news/pfizer-gateway-silent-throughput-bug/", "title": "The bug that halved a gateway's throughput without a single error", "summary": "An engineer wrote a setting that said do not encrypt this connection. The system encrypted it anyway, then sat waiting for a reply that never came. Throughput fell by half, every request still returned success, and no dashboard showed a problem. Pfizer and LiteLLM published the story — and the cause is four lines of code that anyone can read.", "date_modified": "2026-09-16T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "litellm", "gateway", "redis", "observability", "infrastructure" ] }, { "id": "https://development-wec.wiline.com/docs/news/mcp-progressive-discovery-prompt-cache/", "content_html": "![\"Progressive](\"https://development-wec.wiline.com/docs/img/news/mcp-drops-sessions-retires-three-core-features-16x9.webp\")
The MCP maintainers' current roadmap\nnames a problem most people building agents have felt without measuring: a server's\ntool catalogue is charged to the model before anyone asks a question. Under\nImproved primitives, the post is blunt about it —\n\"Connecting to a server with a hundred tools means the model pays for that entire\nsurface before the user has asked a single question, and tool selection tends to get\nworse as the list grows.\"
\nWhat makes this worth reading is not the roadmap. It's that the fix is already\nwritten down, in the client documentation, alongside a warning that it can cost\nmore than the problem it solves — and that warning gets far less attention than\nthe fix it qualifies.
\n\nThe client best-practices page\nthat shipped with the 2026-07-28 specification describes progressive\ndiscovery: the host still calls tools/list as normal, but defers injecting\nthose definitions into the model's context. Instead it hands the model one\nlightweight search_tools meta-tool, and loads full schemas only for what comes\nback.
The page's own diagram puts figures on the difference — roughly 150,000 tokens\nconsumed by definitions alone when everything is loaded upfront, against about\n2,000 under progressive discovery. That is the documentation's own\nillustration rather than a published benchmark, so treat it as the maintainers'\norder-of-magnitude claim, not a measurement you can reproduce. It is still a\nuseful shape: two orders of magnitude, paid before the conversation starts.
\nThe docs also give a threshold rather than leaving it to taste:
\n\n\nImplement a threshold as a percentage of the context window. For example, 1%-5%.
\n
Below that, loading everything is fine. Above it, switch.
\nHere is the sentence that changes how you'd build this. Still on the same page,\nunder Interaction with Prompt Caching:
\n\n\nAdding or removing tool definitions mid-conversation invalidates that cache, and\nthe resulting miss can cost more tokens than the definitions you removed.
\n
Most providers cache the prompt prefix, and the tools array sits near the front\nof it. So the mechanism that saves you 148,000 tokens of upfront definitions works\nby mutating the prefix — which is precisely what a prompt cache cannot tolerate.\nA prefix cache is only valid up to the first thing that changed: edit the array at\nturn six and everything cached after that point goes, which on a long conversation\nis nearly all of it.
The docs' own mitigations are worth reading as design constraints rather than\ntips: append new definitions strictly after the cache breakpoint rather than\nre-sorting the tools array, or route every call through a single stable\ncall_tool({name, args}) meta-tool so the array never changes at all. And treat\ndisconnecting a server as a conversation boundary, not a per-turn operation.
This is where it gets genuinely confusing, and it's worth separating the layers\nbecause they are easy to collapse into one.
\nThe transport cache. tools/list results carry ttlMs and cacheScope\nhints, defined in the specification's caching utility. A client that honours them\nskips the HTTP round trip. This one is MCP's, in the sense that the protocol\nspecifies it.
The host-side memo. Advice rather than protocol — a line in the client\ndocumentation's implementation guidelines, recommending you memoise a fetched\ndefinition so re-injecting it later doesn't need another call. It describes what\na host should do with its own state; nothing crosses the wire. The page is\nexplicit about what it does not do: \"This is separate from what's currently in\nthe model's context.\"
\nThe provider's prompt cache. Not MCP's at all. Owned by whoever serves your\nmodel, keyed on the prefix, and invalidated by exactly the thing progressive\ndiscovery does for a living.
\nWe ran into the first of those the hard way while writing\nPart 3 of the LangGraph series —\na client-side cache=True does nothing unless the server actually advertises a\nTTL, and nothing warns you. Having now read this page properly, the more useful\nlesson is that even getting that right buys you a round trip and nothing else.\nThe schemas still land in context. Those are different problems with different\nfixes, and conflating them is the default mistake.
Part 4 split one MCP server between a\nscheduler and a billing agent by filtering the catalogue per role — a static\nallow-list, decided before the run and fixed for its duration.
\nRead against these docs, that turns out to have a property worth naming: because\nthe list never changes mid-conversation, it doesn't touch the prompt prefix, so\nit sidesteps the cache-invalidation problem entirely. It is a cruder instrument\nthan search_tools — you decide up front instead of letting the model search —\nbut for a small catalogue split across known roles, \"crude and cache-stable\" may\nsimply be the right trade. The docs say as much in the other direction: below the\n1–5% threshold, loading everything is fine.
I would not generalise that further. We have not measured cache-miss cost against\ndefinition cost on the WEC Inference API, and the honest position is that the\ncrossover depends on your catalogue size, your conversation length, and your\nprovider's caching behaviour. What the docs establish is that a crossover\nexists.
\nThe roadmap names five priority areas and commits to no release date or version\nnumber anywhere. It's fair to ask why, and the fairest answer comes from the\nmaintainers themselves. Soria Parra, in the\nMarch roadmap:
\n\n\nA release-oriented roadmap implies a level of predictability that open-standards\nwork rarely has.
\n
That's a reasonable position for a specification developed across working groups, and\nMarch's own record backs it up. \"Transport Evolution and Scalability\" was one of its four\npriority areas, and it named the problem precisely — running MCP at scale had surfaced\n\"a consistent set of gaps: stateful sessions fight with load balancers, horizontal\nscaling requires workarounds\" — while committing to no date for a fix. The July\nspecification delivered that stateless core, five months later.
\nCaching is a different story. It appears nowhere in the March roadmap, and progressive\ndiscovery arrived as client documentation without having been on a roadmap at all — which\nis worth noting before treating either roadmap as a reliable index of what is coming.
\nThe reception hasn't been uniformly warm. The Hacker News thread\non the roadmap ran to 270 points and 161 comments, and the criticism runs in three\ndirections rather than one. The cost of past churn, from colingauvin:
\n\nIt's unreal how bad the initial rollout was between HTTP/streaming and stdio,\nbearer auth and OAuth. Virtually every client/MCP server pair had a different\nportion of that matrix implemented.
\n
The design itself, from nprateem:
\n\nThe real disaster was making it stateful. Need to get some adults in the room.
\n
And whether the protocol earns its complexity at all, from zackify: \"I think the spec\novercomplicates everything honestly.\"
The first of those is the one that bears on this post. It's a fair grievance about\nimplementation drift, and it's the risk to watch with progressive discovery too: this is\ncurrently a client-side pattern with recommended strategies rather than a specified one,\nwhich means two hosts can both be reasonable and behave differently.
\nThe roadmap tells you tool-catalogue bloat is on the maintainers' list. The\ndocumentation tells you what to do about it now, and — to its credit — tells you\nin the same breath that the fix has a bill. If you're loading a large catalogue\ninto every conversation, read that page before you build a discovery layer, and\ncheck where your provider's cache breakpoint sits before you decide the token\nmaths is in your favour.
\n📖 Sources: Model Context Protocol Blog — The New MCP Roadmap · MCP Docs — Client Best Practices · Model Context Protocol Blog — The 2026 MCP Roadmap · Hacker News — New MCP Roadmap
", "url": "https://development-wec.wiline.com/docs/news/mcp-progressive-discovery-prompt-cache/", "title": "MCP's Fix for Bloated Tool Catalogues Has a Bill Attached — And It's in the Docs, Not the Roadmap", "summary": "The MCP roadmap names tool-catalogue bloat as a priority. The client documentation already ships the fix — progressive discovery, with a search_tools meta-tool — and the same page warns that it can cost more than it saves, because loading definitions mid-conversation invalidates the provider's prompt cache. Three layers of caching, only one of which the protocol actually specifies, and the one that bites belongs to your model provider.", "date_modified": "2026-09-07T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "mcp", "agents", "protocols", "context-engineering", "infrastructure" ] }, { "id": "https://development-wec.wiline.com/docs/news/langchain-mcp-first-class/", "content_html": "![\"MCP](\"https://development-wec.wiline.com/docs/img/news/6a99a7d831c3788cf2516870_110.png\")
In July the MCP specification ripped out sessions. We\nwrote at the time that the change was infrastructure, not changelog — that a\nstateless core would let servers sit behind ordinary load balancers and survive\nredeploys, and that clients would need to catch up.
\nToday LangChain caught up. MCP support moved out of the separate\nlangchain-mcp-adapters package and into langchain itself, rebuilt on FastMCP,\nwith two features the old spec made impossible: elicitation as a LangGraph\ninterrupt, and a cacheable tool catalog.
We installed it the same afternoon and pointed it at a server we wrote against the\nnew spec. The announcement is accurate. The ecosystem around it is not ready — and\none of the gaps quietly converts a human-approval gate into a sentence the model\ninvents.
\n\npip install \"langchain[mcp]\", requiring 1.4.0 or newer, and in beta — it says\nso on import, which you can see in our captures further down. Python today,\nTypeScript \"soon to follow.\"
MultiServerMCPClient collapses into\nMCPAdapter. async with MCPAdapter(url) as adapter: then\nawait adapter.list_tools(), and what comes back are ordinary LangChain tools\nthat go anywhere tools go.httpx2.Auth), connection management and protocol negotiation come\nfrom the client underneath. MCP now has two eras — the 2025-11-25 handshake\nprotocol and the 2026-07-28 stateless one — and the FastMCP client picks one\nper connection: \"it tries the new protocol and falls back to the handshake for\na server that hasn't upgraded.\"interrupt(), and you resume\nit with a structured answer. This is only possible because the stateless spec\nturned a mid-call question into a retry-able round rather than something held\nopen on a socket.fastmcp.Client(url, cache=True) honours the server's\nfreshness hints so the tool catalog isn't re-fetched every run.ClientGroup for several servers at once, each keeping its own era and\ncredentials, with tool names prefixed by server — billing_search and\ndocs_search stay distinct.The framing in the announcement is the same one we used in July: \"a redeploy no\nlonger kills live sessions, because there are none.\"
\nOne tool call, before and after, is the clearest picture of what changed — so here it\nis, with a third panel for what we actually got:
\nThe first two panels are the pitch, and the pitch is real. The third is what a\nbrand-new server does on the afternoon the client ships.
\nWe built an MCP server exposing a small booking-and-invoicing database, pointed a\nLangChain agent at it, and gated a refund behind a human. (The agent's own model runs\non a self-hosted gateway — that sits between the agent and the LLM, not between the\nagent and MCP.) It works — that's\nthe tutorial.\nGetting there surfaced three gaps between what's written and what runs.
\nThe first request to a brand-new FastMCP 4.0.2 server — here a plain curl asking\nfor tools/list, so nothing client-side can be blamed for it:
![\"A](\"https://development-wec.wiline.com/docs/assets/images/mcp-session-error-7897aebee33998b591bd27b4606a6e6b.png\")
An error about a concept the specification deleted five weeks ago. The client is\nstateless; the server still defaults to the stateful transport.
\nThe fix isn't a header, a client option, or anything you send. It's an argument on\nthe call that starts your server — the last line of the server file, where mcp\nis your FastMCP instance:
from fastmcp import FastMCP
mcp = FastMCP(\"office-tools\")
# ... your @mcp.tool functions ...
if __name__ == \"__main__\":
# Without stateless_http=True this server answers a 2026-07-28 client
# with \"Bad Request: Missing session ID\".
mcp.run(transport=\"http\", host=\"127.0.0.1\", port=8770, stateless_http=True)
Nothing in the announcement says so — reasonably enough, since the post is about the\nclient. But it means the client half of the upgrade is one pip install, and the\nserver half is a flag you have to know exists.
Two flags exist, and run_http_async (which mcp.run calls for HTTP) accepts both.\nFrom its own docstring:
stateless_http: Whether to use stateless HTTP (defaults to settings.stateless_http)
stateless: Alias for stateless_http for CLI consistency
One switch, two names, and the setting it defaults to is off.
\nctx.elicit is the old API — and the error goes to the model, not to youEvery elicitation example you can find calls await ctx.elicit(message, response_type)\ninside the tool body, where ctx is the Context object FastMCP passes to your tool.\nThat is the handshake-era mechanism: it blocks mid-execution and\nspeaks over the session's back-channel — the back-channel a stateless connection\ndoesn't have. FastMCP's own docs are explicit that it's for connections\n≤ 2025-11-25, and promise that calling it on a modern one \"raises a clear era\nerror rather than failing obscurely.\"
It does raise one. That isn't the problem. We rebuilt the refund tool around\nctx.elicit and ran the same agent against it:
![\"The](\"https://development-wec.wiline.com/docs/assets/images/mcp-elicit-old-api-ec02d789aa6970b70a130f83067c235b.png\")
Read that in order. FastMCP raised the era error, exactly as documented. LangChain\ncaught it and turned it into a ToolMessage with status=\"error\". That is deliberate:\nlangchain/mcp/tools.py wires a handler whose docstring says it exists to hand the\nserver's own error detail to the model \"instead of ending the run.\" The model read\nthat error and told the user:
\n\nI have located Maria Alvarez and identified her open invoice (ID: 2) for $80.00. I\nhave initiated the refund request for this invoice. Please note that a human\nmust approve the amount and provide a reason to complete the refund.
\n
No interrupt was raised. The process exited 0. Nothing is pending, nothing is\nwaiting for a human, and no one will ever be asked — and the invoice is still open,\nso the refund didn't happen either. What you get is not a destructive action slipping\npast a gate; it's an approval workflow that silently doesn't exist, described in\nfluent English by a model that read the error and paraphrased it as progress.
The fix is that the modern pattern has the tool return an InputRequiredResult\ndescribing what it needs, and exit. Your agent surfaces that as the LangGraph interrupt,\na human answers, and the MCP client re-issues the same tools/call with the answer\nattached. But the failure mode is the story: an era error is a fine thing to raise\ninto a program, and a terrible thing to hand to a language model that is rewarded for\nsounding helpful.
cache=True is necessary, not sufficientThe client cache respects the ttlMs and cacheScope hints a server attaches to\nits tools/list response, and only against modern-era servers that send them. A default\nFastMCP server sends neither — we dumped our own tools/list response and it carries no\ncache hints at all. So you turn the cache on, call list_tools(cache_mode=\"use\") twice\nback to back, time both, and get:
![\"Two](\"https://development-wec.wiline.com/docs/assets/images/mcp-cache-bbadf88f6b26a069ec15e3b3db635c02.png\")
You conclude the cache is broken. It isn't — there was nothing to cache. Worth noting\nthe cache belongs to the fastmcp.Client, not to MCPAdapter, and one client per\ncaller keeps catalogs from crossing between tenants.
The elicitation snippet raises AttributeError. The post reads the interrupt as\npaused[\"__interrupt__\"][0].value.requests[0] — attribute access. In\nlangchain/mcp/elicitation.py on 1.4.0:
class MCPElicitationInterrupt(TypedDict):
type: Literal[\"mcp_elicitation\"]
tool_name: str
requests: list[MCPElicitationRequest]
A TypedDict is a dict at runtime, so .requests doesn't resolve. It's\nvalue[\"requests\"] — which the same snippet gets right a few lines later, reading\nquestion[\"key\"] by subscript.
The elicitation docs link 404s. The post closes the section by pointing at\ndocs.langchain.com/oss/python/langchain/mcp/elicitation for \"declining a question, and\ngating destructive tools behind the same approval flow.\" That page returns 404 as of\npublication, while the parent page and its other children — mcp,\nmcp/connections, mcp/tools, mcp/auth — all resolve. It is, inconveniently, the\none page that would have documented the approval flow gap 2 shows falling over.
If you consume MCP servers someone else runs, this is straightforwardly good news and\nyou'll notice mostly the shorter import path.
\nIf you write MCP servers, the July revision moved the ground and the tooling is\nstill settling. Three things are now yours to get right: your server does not become\nstateless because the spec did — you set a flag; a tool that needs human input has to\nbe written to be re-entered rather than resumed — interrupt() unwinds the whole\ncall, so the tool body runs again from the top when you answer; and any\nelicitation example predating August is teaching you an API whose failure lands in the\nmodel's context instead of your logs.
That last one generalises past MCP. As frameworks get better at keeping agents alive\nthrough errors, the class of bug that ends a run is shrinking and the class that gets\nnarrated to a user is growing. A gate that fails closed is a bug you find in testing. A\ngate that was never installed, described by a model as awaiting your approval, is one\nyou find in an audit.
\nThe direction is right. The stateless core is what makes an agent's human-approval pause\nsurvive a redeploy, and that's a real capability rather than a refactor. Just don't\nexpect your first request to succeed.
\n📖 Sources: LangChain — MCP in LangChain: stateless protocol, elicitation, and more · MCP in LangChain docs · Migrating from langchain-mcp-adapters · FastMCP client documentation · FastMCP elicitation · MCP 2026-07-28 specification
\nVersions under test: langchain 1.4.0, fastmcp 4.0.2, mcp 2.1.1, model served over a self-hosted gateway. The ctx.elicit capture is a render of real captured output from that run, not a screen grab.
Your agent can search the web. You write in the prompt: only use sec.gov and the big\nfinancial wires. It usually listens. The times it does not are the times you learn that\na line in a prompt is a request, not a rule.
\nOn 19 August AWS added site filters to the web search tool in Bedrock AgentCore. Small\nfeature. But the way it handles a disagreement is worth knowing, because that is what\nmakes it a guardrail instead of a note.
\n\nThis is AWS's own documentation of an AWS service, quoted here. It does not run on our\ninfrastructure and we have not tested it. The design is what interests us.
Two filters. A list of sites to allow and a list to block, plus a date range for when a\npage was published.
\nThe agent can pass them in the search call itself:
\n{
\"method\": \"tools/call\",
\"params\": {
\"name\": \"WebSearch\",
\"arguments\": {
\"query\": \"latest SEC enforcement actions 2026\",
\"filters\": {
\"domainFilter\": { \"include\": [\"sec.gov\"], \"exclude\": [] },
\"publishedDateFilter\": {
\"from\": \"2026-07-01T00:00:00Z\",
\"to\": \"2026-08-04T23:59:59Z\"
}
}
}
}
}
An admin sets the same kind of list on the gateway instead, where the agent cannot see\nor change it. Each list holds up to 100 sites.
\nSo the admin has a list and the agent has a list. What happens when they disagree?
\n\n\nAllow lists are intersected. Block lists are combined.\n\"Runtime filters can narrow but never expand the scope set by an administrator.\"
\n
The admin's list is the ceiling. The agent can ask for less, never more. And anything\neither one blocks stays blocked.
\nTurn it around and you see why it matters. If the two allow lists were simply added\ntogether, the admin's list would be a starting suggestion. An agent that wanted some\nother site could just name it in its own call and get it. The filter would be paperwork.
\nThat is the difference between the two places you can put a rule. Asking a model to stay\ninside a boundary means asking it to remember, halfway through a job, after reading who\nknows what. A gateway comparing two lists is not remembering anything.
\nThe site list gets the attention. The date range may matter more.
\nAn old page does not look old. A four-year-old page about a tax rule or a dead API reads\nexactly like a current one — same confident tone, and now with a citation stapled to it,\nwhich makes a wrong answer more convincing rather than less. The model cannot judge how\nfresh a page is when nothing tells it. Setting a window gives it a fact it otherwise\nnever had.
\nSearch costs $7 per 1,000 queries and runs in three regions. AWS's selling point is\nthat the queries stay inside their network — \"without sending user prompts and\nretrieval queries to external search API providers outside of AWS.\" Good if you already\nlive in AWS. Mostly irrelevant if you do not.
\nThe idea travels, though, because the tool is reached over MCP: \"Web Search uses a\nbuilt-in connector target on Bedrock AgentCore Gateway using the Model Context Protocol\n(MCP).\" Nothing about allow lists intersect, block lists combine, the agent can only\nnarrow needs Amazon. Any gateway can do it, for any tool — which files something may\nread, which hosts it may reach, which tables it may query.
\nSo the question to ask about a tool is not whether you can restrict it. It is where the\nrestriction lives, and whether the agent can move it.
\nWe went through the protocol under all this in\nMCP's biggest update, and gave a model live search without\na managed service in\nweb search on WEC Inference. The next tutorial\nin the agent series puts an agent's tools behind a gateway that decides what it may\ncall.
\nA model router's job is to read a request and decide which model should answer it.\nCheap questions go to a small model, hard ones to a large one, and the bill comes\ndown. The whole arrangement rests on being able to tell the difference.
\nThen a user types \"yes\".
\nOr \"continue\". Or \"do it\". Nothing in those two or three characters says whether\nthe work being approved is a spelling fix or a database migration. A router\nscoring the current message in isolation sees a very short string with no\ntechnical vocabulary, and does the obvious thing: cheapest model.
\nWhich means if you route requests to save money, your cheapest tier is probably\nabsorbing work it should never have seen — and your savings figure is partly\nfake. On 4 August LiteLLM published a benchmark that measures both halves of\nthat: how wrong the routing gets, and what it costs to fix.
\n\nThe question generalises past their implementation — anything classifying a turn\nin a conversation has this problem — and the reason it's worth reading is that\nthey measured the price of the fix, not only the benefit.
\nEverything below is LiteLLM's own measurement, published on their blog and quoted\nhere: v1.97, classifier gpt-5.4-mini, their three datasets, their reference\nlabels. None of it was run on our infrastructure, and we have not reproduced it.
The sweep: 5,600 live classifier calls against real providers, described as\n\"two sweeps of seven configurations each (classifier_context_window_size of 0,\n1, 2, 3, 5, 8, 10), one with assistant turns in the window and one without, across\nthree multi-turn datasets, with two repeats per conversation.\"
The variable is how many prior turns the classifier gets to see. Zero means it\njudges the current message alone. Ten means it reads the last ten turns first.
\nAgreement with reference tiers, by window size:
\n| Prior turns | Short-reply follow-ups | MT-Bench 2nd turns | ShareGPT multi-turn |
|---|---|---|---|
| 0 | 50.0% | 49.4% | 83.8% |
| 1 | 71.2% | 53.1% | 84.4% |
| 2 | 87.5% | 53.1% | 90.6% |
| 3 (default) | 85.0% | 55.0% | 91.9% |
| 5 | 86.2% | 53.8% | 91.9% |
| 8 | 87.5% | 55.6% | 91.2% |
| 10 | 90.0% | 55.6% | 88.8% |
Figure 1. Agreement with reference tiers by context window, drawn from the figures published in LiteLLM's benchmark. The red line is the 36-follow-up subset whose difficulty only resolves against history.
\nThree different stories in three columns, which is the first useful thing here.
\nShareGPT starts at 83.8% and gains eight points. MT-Bench barely moves at all —\n49.4% to 55.6% across the whole sweep — and their own caveat explains why:\n\"MT-Bench's ceiling reflects its reference labels rather than router behaviour.\"\nThe short-reply set is where it bites, 50% to 90%.
\nInside that first column sits a subset, and it is the sharpest result in the post:
\n\n\nAgreement there is 14% at N=0, 47% at N=1, and 78% at N=2, and flat from\nthere out to N=10.
\n
Fourteen per cent to seventy-eight per cent, achieved by showing the classifier\ntwo prior turns.
\nThat figure describes 36 follow-ups — the ones \"whose final turn only\nresolves against the history\". It is a deliberately selected hard subset, not a\ngeneral accuracy claim, and reading it as \"routers are 14% accurate\" would be\nwrong. What it does show is the shape of the failure: when a turn's difficulty\nlives entirely in what came before it, a context-free classifier is worse than\nguessing, and almost all of the recovery happens by the second turn of history.
\nThe curve going flat at N=2 is the practically useful part. This is not a\n\"more context is better\" result. It is a \"two turns is nearly all of it\" result.
\nThis is the half that makes the finding actionable, and where the honest answer is\nmore interesting than \"it's cheap\".
\nThe window itself is free. Every paired comparison against the zero-window\nconfiguration has a 95% bootstrap confidence interval straddling zero. Window 1\nwith assistant turns off comes in at −17.8 ms, interval −68.9 to +29.6. Their\nexplanation is that the window adds prefill only — the output stays a small fixed\ntier label. Over a 318-to-1,043 prompt-token range, tokens and latency correlate\nat r = 0.007.
\nThe classifier is not free. Adding history costs nothing, but asking a model\nto classify at all costs \"p50 sits near 600 ms in every configuration\" — and\nthat sits in front of the real completion. The classifier here is\ngpt-5.4-mini, and it runs at $0.31–$0.61 per 1,000 requests across the whole\nsweep. Cheap in money, half a second in time.
And accuracy raised the bill. This is the part worth sitting with. For the\nshort-reply set, the modelled routed cost rose with the window — from $2.87 to\n$6.49 per 1,000 requests. Not a regression: the reason is \"a tier mix of 66%\nSIMPLE at N=0 against 30% at N=2\". Two thirds of those follow-ups were being\nserved by the cheapest model. Once the classifier could see what they were\napproving, they stopped being.
\nSo the routing was cheap because it was wrong. The saving was a bill someone else\nwas paying, in answer quality, on requests nobody was auditing. On the other two\ndatasets the effect runs the other way — ShareGPT's modelled cost falls slightly,\nbecause context lets the classifier settle ambiguous prompts into MEDIUM instead\nof defaulting up to REASONING.
\nThat is the useful shape of it: better context doesn't reliably cut spend. It moves\nspend toward where the work actually was.
\nThe default is 3, and 3 is not the best number in any column. Ten is best for\nshort-reply follow-ups at 90.0%, and simultaneously the worst result for\nShareGPT past N=2, dropping to 88.8% from 91.9%. More history is not monotonically\nbetter, and the shipped default is a judgement call across datasets that disagree.
\nAnd they name their own limits, which is the mark of a benchmark worth trusting:\n\"Reference tiers are judgement calls\", \"Routed completion cost is modelled\nrather than billed\", \"One classifier model was swept\", and \"Latency was\nmeasured on a single VM at concurrency 10\". Concurrency 10 on one machine is not\nproduction, and a heavier classifier model carries more absolute latency than the\none they used.
\nThe lesson survives the specific implementation. If you are choosing models per\nrequest — with a heuristic, a classifier, or a hand-rolled rule — the turns that\nwill embarrass you are not the long technical ones. They are the short ones. A\nrouter scoring \"yes\" as a trivial request routes the approval of an architecture\nmigration to the smallest model you own, and nothing in your logs will flag it,\nbecause a cheap answer to a cheap-looking question is exactly what you asked for.
\nTheir fix is a window of prior turns. The cheaper fix, if your classifier has no\nsuch option, is to notice that short replies in an established conversation are\nthe population to worry about, and to stop scoring them on their own contents.
\nWe took the same router apart from the other direction recently — the seven\nscoring dimensions, the arithmetic on a real prompt, and what a free keyword\nscorer misses that a model catches — in\nroute by complexity.
\nAlmost every \"secure your LLM app\" guide gives the same advice: before a prompt reaches\nthe model, strip the personal data out of it — swap names, emails, and account numbers for\n[REDACTED] or <PERSON>, then call the model.
It sounds obviously right. I assumed it was, too. Then I went and read the research on what\nmasking actually does to a model, and the papers point the other way. The short version:\nmask your logs, not your prompts.
\n\nPII is personally identifiable information — a name, an email, a phone number, an\naccount or government ID. Data that points at a specific person.
\nWhen people reach for pre-call masking, they're usually blending two different worries into\none:
\nPre-call masking is aimed at #1. Hold on to that — it turns out to matter.
\nHere's the simplest failure. Put three people in a prompt and redact all of them to\n[REDACTED], and the model can no longer tell them apart. Who signed the contract? Who\nwas cc'd? The words that carried those relationships are gone, so the answer degrades.
This isn't just intuition. A 2026 benchmark called RedacBench measured the trade-off\ndirectly, across 514 texts and 187 policies. Even when a capable model does the redacting,\nturning the security dial up to ~81% of sensitive content removed leaves you keeping only\n37.6% of the text's non-sensitive meaning. You throw away roughly 60% of what made\nthe prompt useful to buy that privacy.
\nThe obvious fix is to stop using dumb placeholders. Deterministic tokenization maps the\nsame value to the same token every time — \"John Smith\" always becomes PERSON_42,\n\"Jane Doe\" always PERSON_17 — so the model can still track who did what. That's genuinely\nbetter.
But it isn't free either. One engineer ran 109 masking tests across healthcare, legal,\nfinancial, and developer workflows and wrote up where it broke. (Full disclosure: he also\nsells a tokenization tool, so take his framing with a grain of salt — but the failures he\nlogged are concrete and easy to reproduce.)
\nGOV_ID_8x3m, but leave the words\n\"social security number\" sitting next to it, and the model's safety filter can refuse the\nwhole request — it sees a sensitive label beside an opaque token and flags it.His overall detection came out to 89% — and his sharper point was that the missing 11%\nis where it hurts: you're forced to either block a legitimate request or leak. There's no\ncomfortable default.
\nStep back and it fits a pattern the research keeps finding: models are fragile to how a\nprompt is worded.
\nNeither of those papers tested PII masking — so this next step is my inference, not their\nclaim — but masking is a prompt edit. It makes the prompt less specific and changes its\nwording, which is exactly the lever these papers show models are sensitive to. You're\nrolling dice you don't need to roll.
\nMasking also costs money in a way that's easy to miss, because it changes the prompt on\nevery call. That quietly breaks prompt caching — the discount you get when a prompt's\nopening is identical to a previous one.
\nBoth major providers cache by prefix, and both say a change up front invalidates it:
\nA cache read costs about 10% of the normal input-token price. So every masked prompt\nthat misses the cache pays close to ten times more on those tokens, and gives up the\nfaster first token too. On top of that, when a masked placeholder like PERSON_42 leaks\ninto a tool call, the tool rejects it and the agent retries — and a study of coding agents\nfound that small prompt-wording changes can multiply token use 2.4–7.4× with no gain in\nsuccess. (Again: not masking specifically, but the same mechanism.)
Here's the part I had backwards. The problem was never that the model sees the data.\nA model reading your prompt to answer it is just doing its job — that inference pass isn't\nwhere data leaks. The real question is retention: what gets stored, and where.
\nOnce you see it that way, the fix is obvious. Put the masking where storage actually happens\nand where you're in control: your logs.
\nThe pattern — often called logging-only — is simple:
\nThink of it as three rungs, worst to best:
\n[REDACTED]) — destroys entity relationships.PERSON_42) — keeps identities, but still triggers refusals and misses.One honest caveat: whatever provider you send prompts to, it's worth knowing its retention\npolicy — that's a separate question from what the model reads, and it's the right place to\nput that worry.
\nThe nice part: this is a configuration, not a rewrite. If you've already\ndeployed a gateway on a WEC Instance, the\nguardrail runs in log-only mode — clean prompt out to the model, masked copy into the logs.\nA follow-up tutorial will wire it up end to end.
\nMasking a prompt before the model buys you a compliance checkbox and quietly sells your\nmodel's intelligence. Put the privacy work where the data actually rests — in the logs — and\nlet the model read the real thing.
\nSomeone posted spec-driven development on LinkedIn this week as the answer to vibe coding —\nto prompting an agent, half-understanding what you're building, and ending up with code you\ncan't vouch for. The linked toolkit has 127,000 stars and comes from GitHub itself. The pitch\nlands.
\nSo I installed it and pointed it at a deliberately trivial task. One of the three principles it\nwrote for me was a dependency policy I never asked for — hold that thought.
\nTwenty minutes isn't a verdict, though. Two engineers have tested this properly, on real\nproblems, long enough for the seams to show. They used different tools, on different\ncontinents, seven months apart — and both reached for the same comparison, unprompted: the last\ntime our industry tried to generate working code from documents. On the one question that\ndecides whether any of this survives contact with AI features, they flatly contradict each\nother. Neither has a measurement.
\n\nIf the term is new to you, the idea is simple. Instead of prompting an agent and iterating until\nthe code looks right, you write a structured specification first — what you're building, why,\nand what \"done\" means — and the agent works from that document rather than from your prompt.\nThe spec, not the code, becomes the thing everyone points at.
\nSpec Kit is GitHub's implementation. It installs into a repo once, not per task, and gives\nyour agent a set of commands: constitution to set project principles, then specify, plan,\ntasks, implement. Nothing runs in the background — it drops templates and command\ndefinitions into your project, and from then on you're talking to your agent. Closer to a linter\nconfig than a product.
Before any of that, though, comes constitution — written first, before anyone has seen the\nproblem, and everything downstream gets judged against it. Hold on to that too.
The setup step tells you what it's really aiming at:
\n![\"Spec](\"https://development-wec.wiline.com/docs/assets/images/spec-kit-agents-f508d78c53fe54f1ad55cd522d110859.png\")
Thirty-plus agents. This isn't a GitHub-only tool — it wants to be the layer every coding agent\nplugs into, which goes a long way to explaining the star count.
\nBirgitta Böckeler, Distinguished Engineer at Thoughtworks, trialled three of these tools by\nhand — Kiro, Spec Kit and Tessl — and\npublished what she found\nin October 2025.
\nShe asked Kiro to fix a small bug. It produced four user stories and sixteen acceptance\ncriteria, including — verbatim — \"As a developer, I want the transformation function to handle\nedge cases gracefully, so that the system remains robust when new category formats are\nintroduced.\" Her summary: \"like using a sledgehammer to crack a nut.\"
\nOn Spec Kit with a real feature — a few days' work, by her own estimate — she never\nfinished the implementation, and reckons she could have built the thing by hand in the time she\nspent reviewing artifacts. The line that will land with anyone who has done a code review:
\n\n\nTo be honest, I'd rather review code than all these markdown files.
\n
She also found the agent ignoring the documents meant to steer it. Spec Kit's research step\ncorrectly catalogued existing classes; the agent then read those descriptions as a\nspecification and generated the classes again, as duplicates. And the reverse failure — the\nagent going \"way overboard because it was too eagerly following instructions (e.g. one of the\nconstitution articles).\"
\nAlex Punnen went narrower and deeper, and\npublished the whole transcript with line citations.\nSpec Kit v0.8.9, on a problem at real scale: querying US elevation data across 1,756 map tiles\n— 23 billion measurements, 180GB.
\nHis finding is subtler than \"the tool invents things.\" He asked for principles covering code\nquality, testing, consistency and performance, and says plainly: \"The principles themselves are\nreasonable.\" What went wrong, he argues, is that one of them quietly tilted every later\ndecision:
\n\n\nNew dependencies MUST be justified in writing: problem solved, alternatives considered,\nlicense verified.
\n
Sensible in isolation. But as Punnen puts it, \"the stdlib option always wins ties because it\ncosts zero justification entries.\" Four phases later the plan chose SQLite because — first\nreason listed — \"Stdlib, zero new dependency… that's the cheapest possible answer.\" Three of\nthe four rejected alternatives fell to that same rule. His verdict: \"The constitution did the\nrejecting; the agent was just the microphone.\"
\nThen comes the part I found hardest to shake. Earlier in the process the agent had written a\nfive-minute performance budget into the spec — a number it guessed, with no measurement behind\nit, filed under the label SC-008. Later, that guess came back as the reason a better design\ncouldn't be used: \"The transcode cost blows past SC-008.\" Only under direct pushback did it\nconcede: \"You're right that I overweighted reason #2.\" The better design, Punnen writes, needed\nnothing that wasn't already in the spec, \"except the willingness to revise an arbitrary number\nthe spec itself produced.\"
\nHis summary of that phase applies to the whole category:
\n\n\nThe artefact looks done because every template slot is filled — not because the engineering\nquestion is answered.
\n
Two things make this more than a one-off. His constitution prompt was essentially GitHub's own\ndocumented example, lightly adapted — he followed the quickstart. And the resistance he hit is\npartly by design: GitHub's methodology document calls the constitution \"a set of immutable\nprinciples,\" with a section headed \"The Power of Immutable Principles.\"
\nTo be precise, the five-minute budget was not a constitutional principle — it was a success\ncriterion the agent generated downstream of one. The document never claims those are immutable.\nBut the number carried that authority anyway, and it took a human to dislodge it. The\nmethodology argues for fixing principles and says nothing about what happens when guesses\nderived from them inherit the same standing.
\nBöckeler's most useful contribution is a distinction the rest of the debate skips. \"Spec-driven\ndevelopment\" covers three different practices:
\nHer verdict on where the tools actually sit: \"All SDD approaches and definitions I've found are\nspec-first, but not all strive to be spec-anchored or spec-as-source.\" Including Spec Kit.\nGitHub's methodology aspires far higher — \"Specifications don't serve code—code serves\nspecifications\" — but Spec Kit creates a branch per spec, so a spec lives for the lifetime\nof a change request, not a feature. Her conclusion: \"spec-kit is still what I would call\nspec-first only, not spec-anchored over time.\"
\nWorth knowing that IBM published the identical three-level\ntaxonomy seven months later,\nfootnoted. If you've seen it credited to IBM, it's hers.
\nI ran Spec Kit at commit 83883a2 on a deliberately minimal prompt — \"Principles for a small\nPython utility. Keep it minimal — I have no strong constraints.\"
![\"The](\"https://development-wec.wiline.com/docs/assets/images/spec-kit-constitution-2e19dee68aab5c7e2a098a49a54edc1b.png\")
One of the three principles it wrote:
\n### II. Minimal Dependencies
Prefer the Python standard library. A third-party dependency MAY be added only when it
removes clearly more complexity than it introduces, and MUST be recorded in the project's
dependency file (e.g. `requirements.txt` or `pyproject.toml`).
A dependency policy, written in the MUST/MAY language of a formal standard, from a prompt where\nI said I had no constraints. It isn't in\nthe local template or the skill file — but Article I of GitHub's nine constitutional articles\ndoes ask for implementations \"with clear boundaries and minimal dependencies.\" So it's\nconsistent with the published philosophy rather than invented on the spot. I can't tell you the\nmechanism, only what went in and what came out.
\nOne run, one trivial task, and it cost me nothing because nothing was at stake. Punnen's case\nshows what this kind of bias costs when the problem is hard enough for it to be wrong. Mine only\nshows it turns up unasked — which matters because, as Böckeler notes, Spec Kit's constitution is\nits memory bank, \"a very powerful rules file\" applied to every change. An unrequested\npreference doesn't sit in a document you'll discard. It becomes a standing rule.
\nHere's what convinced me this is worth taking seriously rather than dismissing or evangelising.
\nBöckeler, who worked on model-driven development early in her career, sees MDD:
\n\n\nI wonder if spec-as-source, and even spec-anchoring, might end up with the downsides of both\nMDD and LLMs: Inflexibility and non-determinism.
\n
Punnen, twenty years in telecom, reaches independently for Rational Rose and UML — \"treated as\nthe silver bullet of its decade: draw boxes, arrows, and diagrams, and the tool would magically\nturn them into working code.\"
\nNeither cites the other. Different tools, different problems, different countries. Both land on\nthe same era: the last time our industry believed a document could be the source and code the\noutput.
\nBöckeler is careful about the comparison — \"I'm not nostalgic about my MDD experience.\" Her\npoint is that today's tools drop the parts that made MDD painful: you no longer need a special\nspec language or a purpose-built generator. What she wonders is whether the exchange is a good\none, since the old approach at least produced the same output every time.
\nPunnen names the trap underneath:
\n\n\nThe specification has to be very rigorous in the first place, but to become rigorous it needs\nto be iteratively refined alongside the generated code.
\n
A Catch-22, in other words: on his account you can't write a rigorous spec for a problem you\ndon't yet understand, and understanding arrives while building. He traces the thought back to\nFred Brooks, forty years ago: \"descriptions of a software entity that abstract away its\ncomplexity often abstract away its essence.\"
\nOn one question these two are in direct opposition.
\nBöckeler, generating code repeatedly from one Tessl spec: \"I have seen the non-determinism in\naction… an interesting exercise to iterate on the spec and make it more and more specific to\nincrease the repeatability of the code generation.\"
\nPunnen: \"This is not a major problem in practice. SDD frameworks act as structured prompts, and\nmodern models produce highly consistent outputs when guided by them.\"
\nGitHub takes Punnen's side and goes further, claiming \"Consistency Across LLMs: Different AI\nmodels produce architecturally compatible code.\" Not the same model twice — different models.\nNo evidence offered.
\nTwo experienced engineers, opposite conclusions, a vendor claim stronger than either, and not a\nnumber between them. It matters more than it looks. Keeping a spec and its code in step — the\nspec-anchored idea — relies on automated tests to catch the drift. That works for deterministic\ncode, where the same input gives the same output.
\nPoint it at an LLM feature and the check stops working. assert response == expected means\nnothing when the response differs every run. Unless you swap tests for evals, where each\nacceptance criterion becomes a scored assertion: did it classify correctly, did it return valid\nJSON against the schema, did it refuse when it should have.
That bridge survives nondeterminism, and it's the harness I've spent several tutorials building\non the WEC Inference API. It also makes the disagreement measurable: write the spec, turn its\ncriteria into eval assertions, and run them across a long session to see whether adherence holds\nor decays.
\nThat's the next post.
\nPunnen's practical takeaways are better than anything I'd invent, and he earned them:
\nOne of my own: write your principles with a date and a rationale, so a later you can supersede\nthem. IBM suggests treating specs as \"stackable versioned artifacts, like architecture\ndecision records\" — and an ADR is something you can mark superseded. The methodology calls\nthese principles immutable. Your problem isn't.
\nAdd IBM's cost test, the sharpest practical line any of them wrote: the cost of refining the\nspec should always be lower than the cost of fixing misunderstandings in implementation. When\nthat balance flips, stop polishing and start building.
\nAnd check the org before you install. The repo doing the rounds on LinkedIn is a fork with\neleven stars; the real project is github/spec-kit. That\nmatters more here than for most tools: Spec Kit's job is writing instruction files your agent\nthen obeys, and on first run you approve that folder in one keystroke without having read them.
So does it replace vibe coding? Not the way the pitch suggests. It doesn't help when you don't\nunderstand your problem — it helps when you understand it and communicate it badly. Those are\ndifferent failures, and only one of them has a template.
\nBoth found real value in the early phases — Punnen rates the clarification step his most useful\nof all — and both found the artifacts looking most authoritative exactly where the decisions\nunderneath them were most arbitrary. Punnen: \"they are at their most dangerous when they look\nthe most rigorous.\"
\nBöckeler reaches for a German compound word for it — Verschlimmbesserung. Making something\nworse in the attempt of making it better.
\nThe hard part was never writing the code.
", "url": "https://development-wec.wiline.com/docs/news/spec-driven-development-solution-to-vibe-coding/", "title": "Spec-Driven Development: is it the solution to Vibe Coding?", "summary": "A GitHub toolkit with 127k stars says you should write the spec before the code, and let the agent build from it. I ran it, then read the two engineers who tested it properly — and both reached for the same comparison: the last time our industry tried generating code from documents.", "date_modified": "2026-08-14T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "engineering-practice", "agents", "evals", "tooling" ] }, { "id": "https://development-wec.wiline.com/docs/news/muse-glimmer-30b-local-agentic-model/", "content_html": "Most \"run it locally\" model announcements come with an asterisk — smaller, weaker, a toy\nversion of the real thing. Meta's newest release doesn't: Muse Glimmer, a 30B\nmultimodal model built specifically for agentic work, fits on a single consumer GPU and\nbeats larger models on the benchmarks that actually measure agent behavior.
\n\nMuse Glimmer is 30B parameters total: a 2B ViT-style vision encoder bolted onto a\n28B-parameter text decoder, 52 transformer layers using a hybrid attention pattern.\nIt's distilled from Meta's larger Muse Spark model — the capability of a bigger model,\ncompressed into something a single GPU can hold. Training data spans 100+ languages,\nwith a January 4, 2026 knowledge cutoff.
\nThe headline feature: multimodal tool-calling with structured outputs. It can look at\nan image and decide which function to call based on what it sees, not just parse text\ninstructions — e.g. inspecting a screenshot and calling the right API based on what's\nrendered, not a text description of it. It also generates and executes code, and is\nbuilt with explicit failure-recovery behavior rather than assuming every tool call\nsucceeds on the first try.
\ntransformers, llama.cpp, vLLM, and Inference Endpoints — no\nwaiting on community ports.Apache 2.0 — no commercial-use gate, no attribution requirement, no separate license\nnegotiation to run it in a product. Worth stating plainly because it's not a given right\nnow: some recent open-weight agentic models carry commercial-use restrictions that only\nsurface once you read the license text closely. This one doesn't.
\nMeta also ran safety evaluations for chemical/biological, cybersecurity, and\nloss-of-control risk — all rated \"moderate or lower.\"
\nMeta's own published numbers, against Gemma4-31B and Qwen3.6-27B:
\n| Benchmark | Muse Glimmer | Gemma4 | Qwen3.6 |
|---|---|---|---|
| MCP Atlas (agentic) | 75.5 | 54.2 | 62.5 |
| DeepSearch QA | 74.6 | 61.7 | 71.1 |
| GAIA2 | 43.3 | 36.4 | 40.0 |
| SWE-Bench Pro | 51.2 | 36.9 | 50.2 |
| SWE-Bench Verified | 76.0 | 66.6 | 77.2 |
It's not a clean sweep — Qwen3.6 edges it on SWE-Bench Verified — but against Gemma4 the\ngap is wide, and against Qwen3.6 it's competitive or ahead on most agentic-specific\nbenchmarks. (Numbers from\nMeta's Muse Glimmer announcement;\nbenchmarks are directional, not gospel.)
\nThe self-hosted AI story has always had a quiet tax: the good agentic models needed real\ninfrastructure, so \"run it yourself\" often meant \"run a worse version of it yourself.\" A\nmodel that's genuinely built agent-first, ships permissively licensed, and fits on\nhardware a single person can own is the gap closing in real time — the same trend this\nwhole tutorial series has been betting on.
\nNot yet — and we're not going to pretend otherwise. We're running it through our own\nmodel-evaluation suite now, the same one that benchmarks everything already on WEC\nModels, before it earns a place in the catalog. If it holds up against what's already\nthere, expect it soon.
\n📖 Sources: Meta's Muse Glimmer announcement (Hugging Face) · Model card
", "url": "https://development-wec.wiline.com/docs/news/muse-glimmer-30b-local-agentic-model/", "title": "Muse Glimmer: a 30B agentic model that runs on one GPU, no data center required", "summary": "Meta released a 30B multimodal model built for local agent workloads — beats Gemma4 and holds its own against Qwen3.6 on agentic benchmarks, and fits on a single consumer GPU. Here's what's actually new, and what it'd take for it to land on WEC.", "date_modified": "2026-08-10T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "models", "open-weight", "agents", "local-inference" ] }, { "id": "https://development-wec.wiline.com/docs/news/loop-engineering-graph-engineering-what-survives/", "content_html": "In June 2026, the AI world got a new buzzword: loop engineering — roughly, disciplined\ndesign of a single agent's tool-calling loop. Six weeks later it was supposedly replaced by\ngraph engineering — wiring up several agents at once — killed by twelve words that 3.1\nmillion people saw:
\nAre we still talking loops or did we shift to graphs yet?
— Peter Steinberger 🦞 (@steipete) July 18, 2026
Don't know what loop engineering is? Don't worry — neither did most of the people\ndeclaring it dead. Here are both terms, how a name became an obituary in six weeks, and\nthe twist nobody checked before writing about it.
\n\nThe split shows up best on a job with independent work that can run at once and where a\nwrong answer is expensive — a research briefing: \"Pull together what we know about X —\nthe open web, our own docs, and the repo — and give me a one-page brief with sources.\"
\nThe OpenClaw agent you stand up in these\ntutorials works the loop way — one agent cycling through its tools, one turn at a time.\nGraph engineering wires up a team of specialists instead. Same job, two shapes:
\n![\"Loop:](\"https://development-wec.wiline.com/docs/assets/images/loop-vs-graph-light-d1fd8017587aa7c27d81ad36a7ec1f0b.png\")
![\"Loop:](\"https://development-wec.wiline.com/docs/assets/images/loop-vs-graph-dark-f9ef88bb5be59e4d6a1396b3985067fd.png\")
Read the loop on the left as a wheel: one agent visits each tool, then comes back\naround. Push it past simple tasks and two cracks appear: it's slow (three independent\nsearches still run back-to-back, one worker at a time), and it's hard to trust (the same\nagent searched, read, and wrote the brief, so a wrong claim has no address — you can't tell\nif the search was thin or the agent invented it).
\nThe graph on the right fixes both: specialists split the job, a planner routes it. The\nsearches now fire at the same time — the brief lands in the time of the slowest one, not\nthe sum — and each agent owns one source, so a bad claim has an address. A dedicated\nfact-check node, the step a rushed loop skips, becomes a real gate.
\nNone of that is free: one prompt becomes a planner, four agents, and a verifier — a new\nfailure mode where the coordination itself can break. Trust problem traded for a plumbing\nproblem. And every box in the graph still runs its own loop inside: a graph contains\nloops — that's the whole point.
\nLoop engineering is about one agent. An AI agent is just a model in a while loop\nwith tools: give it a goal, it picks a tool, your code runs it, the result goes back in,\nrepeat. ChatGPT answering a question is one call. An agent that edits a file, runs the\ntests, sees them fail and tries again — that's the loop, five times over. Loop engineering\nis designing that cycle deliberately: what triggers it, who verifies the work, when it\nstops, what happens on failure. The slogan is a good one — the intelligence lives in the\nmodel, the reliability lives in the loop.
Graph engineering is about several agents. Nodes are agents, edges are dependencies,\nplus the shared state and permissions between them. If loop engineering is \"make one agent\nreliable,\" graph engineering is \"make ten agents not step on each other.\"
\nDifferent problems, different scale. So \"loop engineering is dead, graph engineering\nreplaced it\" was never a sensible sentence — it's like saying wheels replaced cars.
\nThe name for the first one arrived in June 2026, from\nAddy Osmani. It caught on fast\nbecause everyone was already doing it, badly, with no shared vocabulary. Remember that\ndate.
\nHere's the thing nobody checked before writing a guide: \"graph\" isn't a 2026 coinage, and\nthe receipts are sitting in a free MIT textbook. From Mathematics for Computer Science\n(6.042J), chapter 6, page 189 — Definition 6.1.1:
\n\n\n\"A directed graph G = (V, E) consists of a nonempty set of nodes V and a set of directed\nedges E… A directed graph is simple if it has no loops (that is, edges of the\nform u→u) and no multiple edges.\"
\n
The formal definition of a graph already contains the word loop, as an ordinary feature\nof graphs — not their opposite. Definition 6.1.2 adds that a cycle is a walk that\nreturns to where it started, which is what a loop is.
\nA loop isn't the opposite of a graph. It's a graph that comes back to an earlier node.\nMathematics has known this since Euler walked around Königsberg in 1736.
\nThe fuse wasn't an engineering discovery — it was two product launches colliding.\nDeepLearning.AI released a course on knowledge graphs with multi-agent systems, taught by\nNeo4j's Andreas Kollegger. Around the same time, Linear shipped an agent feature called —\nof all things — Loops. Two companies, two unrelated products, two words that sounded\nlike rival philosophies.
\nThen Steinberger, who built OpenClaw, posted\nhis twelve words at 9:34 PM on July 17. Four and a half hours later Hamel Husain\npublished an X Article titled \"Loop Engineering Is Dead. Enter Graph Engineering,\" and\nSantiago Valdarrama picked it up the\nsame day. Within 48 hours the new term had three competing definitions and a wave of\ncopycat posts.
\nNeither founding post was serious — the writers who tracked the cycle say so outright.\nLouis-François Bouchard put it plainly: \"my whole feed decided we have a new discipline.\nTo be honest, both tweets were jokes.\" Husain's own follow-up only widened the wink,\npromising the piece was \"not what you think it is.\"
\nAnd almost nobody could check: the article sat behind X's Premium paywall. A paradigm's\nfounding text was something few of the people citing it had actually read. What spread\nwasn't an argument — it was a shape: a punchy title, a wink right behind it, and an industry\nthat answered a joke by writing documentation for it.
\nThe sharpest response came from outside the agent crowd. David Khourshid created\nXState — a widely-used library for modeling exactly this kind of state machine in\ncode — and has been modelling these structures for a decade:
\nFirst it was loops. Now it's graphs. Next month it'll be something else. Here's the thing: we're constantly rediscovering decades-old software engineering patterns and repackaging them as innovations or whatever, instead of just applying what we've already known for a long time.
— David Khourshid (@DavidKPiano) July 20, 2026
His explanation takes two minutes. A state machine answers one question — given the\ncurrent state, when an event occurs, what is the next state? Draw it and states become\nnodes, transitions become edges. Then he turns it on the July argument:
\n\n\n\"Surprise… loops are graphs: directed, cyclic ones.\"
\n
A loop, he notes, is the most basic state machine there is: two states, looping and\ndone. He posted a diagram of it captioned \"This is the silly thing you all hyped for\nweeks.\"
That's the object six weeks of discourse was about. And real agents — the ones\nthat retry, backtrack, wait for a human — were never sequential and never acyclic to begin\nwith (\"sorry, DAG lovers\").
\nThe creator of LangChain got there from the opposite direction, his LangGraph being the\nimplementation everyone kept pointing at:
\nSo i didn't really know what graph engineering is, and i still don't really… but it's basically just langgraph?
— Harrison Chase (@hwchase17) July 20, 2026
Four days later he and Sydney Runkle published a longer answer,\n3 Years of Graph Engineering with LangGraph,\nwhich is the most useful thing written during the whole episode. Their opening line is\nthe best description of the phenomenon I've read:
\n\n\n\"It's the latest term to come out of X's AI content factory, joining prompt engineering,\ncontext engineering, harness engineering, and loop engineering.\"
\n
And then, from the people whose product is literally the graph:
\n\n\n\"Loops are simple graphs. Loop engineering isn't an alternative to graphs, so much as a\nsimple version of them.\"
\n
They also confirm the thing most July posts got backwards: production agent graphs are\nusually not DAGs. Real agents retry, ask for missing information, revise after\nvalidation, pause for a human. Cycles aren't a design flaw to be engineered out — they're\nthe job.
\nThe clearest framing of all, though, came from a reply:
\nGraph engineering is deciding where the work is allowed to go. Loop engineering is making the work get better each time it runs. Graph is the rails. Loop is the motor. Rails keep you from crashing. The motor is what actually moves.
— Eric Osiu (@ericosiu) July 22, 2026
It's August 5, about two and a half weeks after the obituary. Nobody won. The debate\ndidn't resolve and it didn't die — it got absorbed by content marketing. Every\nagent-infrastructure vendor now has a \"definitive guide to graph engineering,\" and behind\nthem a long tail of SEO pages saying the same thing in the same order.
\nNothing was learned in that time. No benchmark ran, no production system proved one\napproach beat the other. Two companies shipped unrelated products, two well-followed\ndevelopers made a joke, and a lot of people agreed on a word — then agreed on a different\nword.
\nThat has a real cost. If you tried to keep up by adopting each term as it trended, you\nrewrote your architecture twice in July for reasons that were social, not technical.\nMeanwhile the engineer who ignored both posts and spent that month adding stop rules and\ntyped state came out ahead, because those mattered under any label.
\nOne idea from this month deserves more attention than the naming war, and it comes from\nTowards AI's breakdown:\nmore agents doesn't mean more judgement.
\nTwenty agents running the same model, reading the same flawed context and checking against\nthe same broken metric will agree with each other at industrial scale. Worse is the\ncircular version: one agent checks a report against another report, an audit agent checks\nboth against a dashboard, and the dashboard was built from the same data. Every node\nagrees. Nothing touched reality. The system looks well governed, because the diagram has\nreviewers everywhere.
\nThe fix is what they call reality anchors — evidence from outside the agent system.\nTests that actually ran. Money that reached the bank. Customers who stayed. Rules the\noptimiser can't quietly rewrite. Their line is the one I'd put on the wall:
\n\n\n\"Without anchors, a graph is a larger hallucination with better project management.\"
\n
The same trap exists one level down, inside a single loop: if the agent that does the work\nalso decides whether the work is good, you've built an expensive machine for agreeing with\nitself. A verifier only counts if it can actually fail you.
\nKhourshid's line is the filter, and it isn't cynicism: next month it'll be something\nelse. When the next term lands, the question is never \"is this the new paradigm?\" It's\n\"what specific failure does this name, and do I have that failure yet?\"
\nDiagnose where your bottleneck actually is:
\nAnd a case for neither: if the task is genuinely open-ended — research, exploration —\nforcing it into fixed paths is the wrong move. The LangChain team makes this point against\ntheir own product: they built early deep research on predefined LangGraph workflows and\nthen moved to a looser agentic loop, and GPT Researcher did the same. Structure you\nhaven't earned costs you.
\nAnd whichever you have, these outlive the vocabulary: every loop needs a stop rule; state\nneeds a shape and save points; acting nodes need permission boundaries and a human gate on\nanything irreversible; add complexity only when a real failure asks for it.
\nIf you want the two concepts that pay off across all of it, take Khourshid's\nrecommendation over any of July's vocabulary: state machines and the actor model. Both\npredate this argument by decades and will outlive whatever replaces it — probably in about\nsix weeks.
\nThe Model Context Protocol just had its biggest revision since launch. The\n2026-07-28 specification doesn't add a feature — it rewrites how every MCP\nserver talks to every client. If you've deployed an MCP server anywhere past\n\"runs on my laptop,\" this one touches your infrastructure, not just your\nchangelog.
\n\nThe headline change: MCP is now stateless at the protocol layer. Every\nrequest carries its own protocol version, client identity, and capabilities —\nthere's no more initialize/initialized handshake, no session ID, no\nrequirement that request N+1 lands on the same server instance that handled\nrequest N. Lead maintainer David Soria Parra called it\n\"a leap in serving scalable MCP servers\",\nbuilt on 18 months of running MCP past the local-tool stage.
That one change unlocks a chain of practical ones:
\nMcp-Method and Mcp-Name\nHTTP headers, so gateways and firewalls can route and rate-limit by\ninspecting headers instead of parsing every JSON body.tools/list, prompts/list, resources/list,\nand resources/read now return ttlMs and cacheScope, so clients know\nhow long they're allowed to skip re-fetching.resultType: \"input_required\";\nthe client retries the same call with inputResponses filled in — no\npersistent connection required.State didn't disappear, it just became explicit: if your tool genuinely needs\nit, it mints a handle and hands it back to the client to pass in on the next\ncall, instead of hiding it in the transport layer.
\nThis is the part that should get an AI engineer's attention before the\nprotocol change does. MCP authorization now aligns with OAuth 2.1 and OpenID\nConnect instead of leaving it to each implementer to wire up their own\nversion, as WorkOS breaks down:
\niss\nparameter before redeeming a code, closing the authorization-server\nmix-up class of bugs.The practical upshot: the \"confused deputy\" problem — a tool call executing\nwith credentials meant for a different server — gets closed at the protocol\nlevel instead of depending on every server author to remember to check.\nIf you're running multiple MCP servers behind one gateway (which, per our own\ngateway coverage, is where this is heading for\neveryone), this is the part of the update that actually reduces your risk\nsurface, not just your ops burden.
\nNothing here is backward compatible by accident. Soria Parra, in\nThe Register's reporting,\ndidn't sugarcoat it: \"If you built your own implementation, it's going to be\na lot of uplift to make this correct,\" and the stateless redesign, by his own\nadmission, \"makes things on the wire a bit more complicated than they used\nto be\" even as it removes session state. Roots, Sampling, and Logging are\nnow formally deprecated — Sampling for confusing semantics, Roots as \"a very\nniche thing,\" Logging for being excessively verbose — with a twelve-month\nminimum window before they're actually removed, alongside the legacy\nHTTP+SSE transport.
\nRead charitably, this is a protocol growing up: a formal deprecation policy\nmeans you get a year of notice instead of a surprise break. Read skeptically\n— and The Register does — this is fixing problems that only showed up once\nMCP left local dev tooling for cloud deployment, which says something about\nhow much load-bearing infrastructure got built on the earlier design before\nanyone stress-tested it at scale. Both readings are true at once. That's not\na reason to panic; it's a reason to actually read the migration guide before\nyour integration tests do it for you.
\nIf you're building agents against MCP servers you don't control, you likely\nnotice nothing immediately — Tier 1 SDKs (TypeScript, Python, Go, C#, with\nRust in beta) handle the negotiation. If you run an MCP server — for a\nRAG pipeline, an internal tool bridge, anything past a demo — this changes\nthree things you own directly: how it scales (stateless means your ops story\ngets simpler), how it's secured (OAuth 2.1 alignment means less of your own\nauth code to get wrong), and your clock (twelve months to move off Roots,\nSampling, Logging, and SSE transport before they're gone). None of that is\noptional just because you didn't ask for the rewrite.
\n📖 Sources: Model Context Protocol Blog — the 2026-07-28 specification · WorkOS — authentication changes in the MCP 2026-07-28 spec · The Register — MCP breaks with its stateful past · VentureBeat — MCP's biggest update, what changes for AI agents
", "url": "https://development-wec.wiline.com/docs/news/mcp-2026-07-28-spec/", "title": "MCP Just Shipped Its Biggest Update Ever — Here's What Actually Changes for AI Agent Engineers", "summary": "The 2026-07-28 MCP specification rips out sessions and rewrites authorization. If you build or run MCP servers, this changes your infrastructure, your auth flow, and your deprecation clock — whether you asked for it or not.", "date_modified": "2026-07-28T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "mcp", "agents", "protocols", "infrastructure" ] }, { "id": "https://development-wec.wiline.com/docs/news/gpt-5-6-token-economics/", "content_html": "OpenAI shipped GPT-5.6 on July 9 — a family of three models (Luna, Terra, Sol) — and the\nheadline claim isn't a leaderboard score. It's an efficiency number: frontier coding performance\non less than half the output tokens. If you build agents, that's a claim about your bill,\nnot about bragging rights. It's also exactly the kind of claim you should measure yourself.
\n\nThree tiers, cheapest to strongest, available in ChatGPT, Codex, and the OpenAI API:
\n| Model | Positioning | Input / Output (per 1M tokens) |
|---|---|---|
| Luna | fastest, budget tier | $1 / $6 |
| Terra | mid tier — \"performance competitive with GPT-5.5\" | $2.50 / $15 |
| Sol | flagship, \"best coding model yet\" | $5 / $30 |
All three variants ship native tool use and multimodal reasoning, with long-context evals run\nout to 1M tokens; Sol also powers the new ChatGPT Work professional tier. The claims worth\nknowing (from OpenAI's announcement — directional, not\ngospel): Sol scores 80 on the Artificial Analysis Coding Agent Index, 2.8 points above\nAnthropic's Fable 5 — while using less than half the output tokens, in less than half the time,\nat about a third of the estimated cost — and 88.8% on Terminal-Bench 2.1. On Agents' Last\nExam, an eval of long-running professional workflows across 55 fields, they report Terra and\nLuna beating Fable 5 at around one-sixteenth the estimated cost. Sam Altman's framing: Sol\nis \"54% more token-efficient\" on coding tasks. The release also leans hard on cybersecurity\n— \"frontier performance with significantly fewer tokens.\"
\nTwo API features ship under the same efficiency banner, and they're the most developer-relevant\npart of the launch: Programmatic Tool Calling, where the model writes and runs small\nin-memory programs that coordinate tools and process intermediate results — instead of passing\nevery tool response back through the model, so tool-heavy tasks burn fewer tokens and fewer\nround trips — and a multi-agent beta in the Responses API (the new Sol Ultra setting runs\nfour agents in parallel by default).
\nOne more first, buried in the timeline: GPT-5.6 was planned for June and shipped three weeks\nlate because a US government review gated the release — Commerce's Center for AI Standards\nand Innovation ran additional testing before OpenAI got permission for a public rollout. More on\nwhy that matters below.
\nFor chat, output tokens are a rounding error. For agents, they are the bill: an agentic\ncoding session burns tokens on every step — plans, diffs, retries, tool calls — and output\ntokens cost 5–6× input tokens on every pricing card above. A model that solves the same task on\nhalf the output tokens would be effectively half price and twice as fast at equal quality,\neven if it's only a couple of benchmark points better. If it holds on your tasks — and that\n\"if\" is the whole subject of the next section — that's a real and welcome shift in what vendors\ncompete on.
\nThat's why \"54% more token-efficient\" is a more aggressive competitive move than any leaderboard\njump. And OpenAI wasn't alone — the whole frontier spent this week competing on price per task:\nMeta launched Muse Spark 1.1 for agentic coding at $1.25/$4.25 per 1M tokens (with $20\nfree credits per account), and SpaceXAI released Grok 4.5 — co-trained with Cursor — at\n$2/$6, marketed as roughly 6× cheaper than comparable frontier models. Even OpenAI's\ninfrastructure news pointed the same direction: engineers reportedly halved inference costs\nthrough software optimization alone. The race has visibly changed lanes, from benchmark points\nto cost per task.
\nThe three-tier ladder matters for the same reason. The emerging pattern is routing by task\ndifficulty: cheap tier for classification and extraction, mid tier for everyday generation,\nflagship only for the hard multi-step work. If you run a gateway (we've\nwritten about why that layer matters), this is what it's for.
\nHere's the thing about \"54% more efficient\" and \"2.8 points above\": those numbers come from the\nvendor, measured on the vendor's chosen benchmark, with the vendor's harness. That's not an\naccusation — the numbers are very likely real on that benchmark, and we apply the same\ndiscount to everyone's, including the open-weight models we like (we said exactly this about\nGLM-5.2's tables). It's a structural point: no vendor benchmark can know your workload.
\nYou don't even have to leave OpenAI's own coding table to see it — and credit to them for\npublishing the mixed rows rather than only the flattering ones:
\n| Coding eval (one vendor, one table) | GPT-5.6 Sol | Claude Fable 5 | Who leads |
|---|---|---|---|
| Artificial Analysis Coding Agent Index v1.1 | 80 | 77.2 | Sol, +2.8 |
| Terminal-Bench 2.1 | 88.8% | 83.1% | Sol, +5.7 |
| DeepSWE v1.1 | 72.7% | 69.7% | Sol, +3.0 |
| SWE-Bench Pro | 64.6% | 80% | Fable 5, +15.4 |
(Numbers from OpenAI's own announcement.)
\nFour coding benchmarks, and \"which is the better coding model\" flips depending on the row — with\nthe single largest gap pointing the other way. That's not a knock on either model or on the\ntable; it's what benchmarks are. If one vendor's own page can't produce a single answer, a\nlaunch-day headline certainly can't tell you what happens on your codebase. Token efficiency\ncompounds the problem: it varies wildly by task shape — a model that's terse on Python refactors\ncan be verbose on SQL or long-form answers, and an agent harness different from the vendor's can\nerase (or amplify) the whole advantage.
\nThe good news: this is a solved problem, and you already have the tooling if you followed our\nevals series. The same Promptfoo setup from\npart 1 compares any two OpenAI-compatible\nendpoints on your prompts — with cost and latency assertions, not just quality grades.\nAnd the head-to-head worth running this week isn't GPT-5.6 against its own launch table — it's\nGPT-5.6 against the strongest open-weight model you can serve yourself:
\nproviders:
- openai:chat:<new-model-id> # the challenger everyone's talking about
- id: openai:chat:gemma4 # the open-weight model already on WEC
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
defaultTest:
assert:
- type: latency
threshold: 5000
- type: cost
threshold: 0.002
Same prompts, both models, and the eval reports quality, tokens, latency, and cost side by side.\nAn afternoon of this tells you what no launch post can: whether the efficiency claim survives\ncontact with your traffic. (And if the model backs a RAG service or agent, gate it like we\ngated ours in the capstone — judge for\nsemantics, deterministic regressions for known bugs.)
\nTwo things happened around this launch that matter more together than apart.
\nFirst, the government gate: for the first time, a frontier model's public release needed a\nfederal review to proceed — and the concern wasn't abstract. The same week, Sysdig documented\nJadePuffer, the first fully autonomous AI-agent ransomware operation, which exploited a\nLangflow CVE and then performed reconnaissance, lateral movement, and extortion on its own,\nrecovering from failed attempts within seconds. Whatever your politics, the engineering fact is\nthat access to closed frontier models now has one more valve you don't control — alongside\npricing, deprecations, and rate limits. We made this argument when an open-weight model cracked\nthe proprietary top 10; this release made it for us.
\nSecond, the open-weight world had a loud week too: Google released Gemma 4, an\nopen-weight, natively multimodal family from 2.3B to 31B parameters (dense and MoE variants,\nvision and audio input, a thinking mode); Mistral opened early access on a new open-weight\nMixture-of-Experts family aimed squarely at the frontier gap; and Together AI closed an\n$800M Series C at an $8.3B valuation on the back of open-model inference — citing over\n$1B a year in bookings. The money is saying the same thing the GPT-5.6 delay is saying: models\nyou can download and run are a hedge worth paying for.
\nThe developer takeaway is the same one that's held all series: keep model choice a config\nchange. Code against the OpenAI-compatible API, put a gateway in front, and swapping models —\nclosed, open, or whatever lands on the WEC Models catalog\nnext — is a base-URL and model-ID edit, not a rewrite. We don't offer GPT-5.6 on WEC today;\nthe point is that if you build portable and eval before you migrate, that fact constrains you\nexactly zero.
\nBecause here's the trap in every launch week: new starts feeling like a reason. It isn't —\nit's a hypothesis. The question your eval should answer is whether the shiny closed model beats\nan open-weight model you control — on your prompts, at your latency budget, per dollar — by\nenough to be worth the valve someone else's hand is on. Sometimes it will, and then you switch\nwith evidence instead of hype. And sometimes the open model holds the line on the tasks you\nactually run, and you just saved yourself a migration and a dependency in one afternoon.
\nYou can run that experiment today, because\nGemma 4 — the open-weight, natively\nmultimodal family Google shipped this same week — is already live on WEC inference. And it's\na serious counterpart, not a consolation prize. What Gemma 4 brings to the table:
\nOne model-ID swap and you're on it:
\ncurl https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{ \"model\": \"gemma4\", \"messages\": [{\"role\":\"user\",\"content\":\"Summarize this incident report…\"}] }'
GPT-5.6's token-efficiency push is good news for everyone — even if you never send OpenAI a\nsingle request, it drags the whole market toward pricing honesty. Take the positive at face\nvalue, take the numbers as hypotheses, and let your own eval — GPT-5.6 on one side, Gemma 4 on\nWEC on the other — make the call.
\n📖 Sources: OpenAI — GPT-5.6 announcement · Google DeepMind — Gemma 4 · TechCrunch — OpenAI launches GPT-5.6 · CNBC — public release after government limits · Axios — GPT-5.6 and ChatGPT Work · Engadget — rollout timing · TechTimes — Mistral's open-weight MoE early access · Asanify — Together AI's $800M round · NLPlanet — weekly AI newsletter, July 13 (Muse Spark 1.1, Grok 4.5, Gemma 4, JadePuffer, inference-cost items)
", "url": "https://development-wec.wiline.com/docs/news/gpt-5-6-token-economics/", "title": "GPT-5.6: OpenAI's new pitch is cheaper per task, not just smarter — verify it on your workload before you switch", "summary": "GPT-5.6 (Luna, Terra, Sol) ships with a claim aimed straight at your bill: frontier coding scores on half the output tokens. What that means for agent economics, why you should verify it on your own workload — and how to build so model choice stays a config change.", "date_modified": "2026-07-13T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "models", "openai", "token-efficiency", "evals" ] }, { "id": "https://development-wec.wiline.com/docs/news/glm-5-2-open-weight-top-10/", "content_html": "Look at almost any current model leaderboard and the top is a wall of Anthropic and\nOpenAI. Then, sitting in the top 10, there's one outlier that isn't proprietary at\nall: GLM-5.2 from Z.ai — open weights, MIT-licensed. That's the story worth\npaying attention to.
\n\nOn the Arena.ai agent leaderboard, GLM-5.2\n(Max) lands at #10 — the only open-weight model in the top 10, surrounded\nentirely by closed frontier models from Anthropic and OpenAI. (Leaderboards move;\nthis is a snapshot — check the live ranking.)
\n![\"GLM-5.2](\"https://development-wec.wiline.com/docs/assets/images/glm-5-2-leaderboard-3f2c9fe86e103693aa80fdbcda6b054b.png\")
That's the headline: not that it tops the chart, but that an MIT-licensed model you\ncan download, self-host, and ship commercially is now trading blows with models you\ncan only rent.
\nMax for hard multi-step coding, High for lighter, faster work.Z.ai's published benchmarks put GLM-5.2 shoulder-to-shoulder with the closed frontier on coding, and ahead on some reasoning:
\n| Benchmark | GLM-5.2 | Claude Opus 4.8 | GPT-5.5 |
|---|---|---|---|
| SWE-bench Pro | 62.1 | 69.2 | 58.6 |
| Terminal-Bench 2.1 (best harness) | 82.7 | 78.9 | 83.4 |
| FrontierSWE (dominance) | 74.4 | 75.1 | 72.6 |
| AIME 2026 | 99.2 | 95.7 | 98.3 |
It edges Opus 4.8 on Terminal-Bench, beats GPT-5.5 on FrontierSWE, tops both on AIME, and trails Opus on SWE-bench Pro — remarkably close for a model you can simply download. (Numbers from Z.ai's GLM-5.2 benchmarks; benchmarks are directional, not gospel.)
\nThe gap between open-weight and proprietary frontier models has been closing all\nyear. What's changed is the terms: with an MIT license and a clean API, GLM-5.2\nis something you can own and deploy, not just call. When access to closed models\ncan shift with export controls or pricing overnight, an open-weight model that holds\ntop-10 quality is a foundation that stays put.
\nYou don't need a third-party account to try it — GLM-5.2 is available on WiLine\nEdge Cloud through WEC Models, our OpenAI-compatible inference. Point any compatible\ntool at the WEC inference endpoint and use GLM-5.2 as the model:
\ncurl https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{ \"model\": \"glm-5.2\", \"messages\": [{\"role\":\"user\",\"content\":\"Refactor this for performance…\"}] }'
If you followed the Self-hosting OpenClaw series, this is the natural\nnext move: keep your agent, swap the model — point OpenClaw at GLM-5.2 on WEC instead\nof a closed provider, and you're running a top-10 model you fully control.
\n📖 Sources: Arena.ai agent leaderboard · GLM-5.2 on Hugging Face · Z.ai model docs · GLM-5 technical report (arXiv)
", "url": "https://development-wec.wiline.com/docs/news/glm-5-2-open-weight-top-10/", "title": "GLM-5.2: the only open-weight model in the top 10 — and you can run it on WEC", "summary": "GLM-5.2 is the lone open-weight, MIT-licensed model holding its own against the proprietary frontier — a 1M-token context and top open-source coding scores. And it's available on WiLine Edge Cloud.", "date_modified": "2026-06-24T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "models", "open-weight", "glm", "inference" ] }, { "id": "https://development-wec.wiline.com/docs/news/litellm-rust-gateway/", "content_html": "![\"LiteLLM](\"https://development-wec.wiline.com/docs/img/news/litellm-rust.webp\")
The AI ecosystem is quietly going through the same transition web infrastructure went through years ago: the performance-critical pieces are moving off interpreted runtimes onto systems languages like Rust. LiteLLM rewriting its AI gateway in Rust is the clearest evidence yet — and a sign the AI stack is maturing from experiment into production infrastructure.
\n\nLiteLLM is the open-source proxy a lot of teams put in front of their models to get one OpenAI-compatible endpoint across 100+ providers. If you've run one in production, this line from the announcement will feel familiar:
\n\n\nUnder real load, CPU and memory climb with concurrency, and pods get OOM-killed at the worst time.
\n
That's the quiet tax of a gateway: it sits on the hot path of every request — every completion, embedding, moderation call, and agent action flows through it — so its own overhead and memory footprint multiply across pods and regions. For years AI conversations were about model quality. As teams ship agents, RAG, and multi-model routing to production, the layer in front of the model is turning into a first-class infrastructure concern. Moving it to Rust is what that realization looks like in code.
\nFrom LiteLLM's published benchmarks (reproducible — the harness ships with the post):
\nThis measures the gateway forwarding path (transform → forward → handle response), not a full production workload — but that's exactly the layer you don't want eating CPU and memory under concurrency.
\nMost readers will fixate on 150× lower overhead. But for anyone operating a gateway, the more consequential number is 11× less memory: 359 MB → ~32 MB. Latency is a per-request improvement; memory is what drives your bill and your reliability.
\nA gateway that holds ~32 MB instead of ~359 MB changes the operational math across the board:
\nWhen a component sits on the hot path of every request, shaving an order of magnitude off its memory compounds at scale far more than the headline latency figure.
\nThis is not a v2 and not a rewrite you have to migrate to. Config files, database schema, client APIs, and provider coverage stay the same. They're moving it in careful stages — a pure-Rust core via PyO3 bindings first (data transformation, no I/O), then the full server on axum/hyper — each route shipped to production behind passing parity tests before the next one starts:
\nBeta signup is open now; the roadmap targets OCR routes by mid-August 2026, /chat/completions and /messages by September, and the full server by December 1, 2026.
For developers building on WiLine Edge Cloud, the gateway sits directly between your applications and your models — so a leaner, faster gateway flows straight through to the apps you ship:
\nIf you followed the OpenClaw series, you already put a gateway-shaped thing on the critical path — a reverse proxy, a model router, an agent runtime. The lesson generalizes.
\nFor years, most AI discussion focused on model quality. But as organizations deploy agents, retrieval systems, and multi-model workflows in production, the layer in front of your models is infrastructure — and every millisecond and megabyte on the hot path compounds at scale. LiteLLM's move to Rust reflects a broader industry realization: infrastructure efficiency is no longer a footnote to model performance, it's part of it.
\nWorth watching, not yet worth switching: it's beta, and the Python proxy isn't going anywhere. But the direction of travel is clear.
\n📖 Read the full announcement — the benchmarks, the route-by-route migration plan, and the architecture diagrams are all worth your time: LiteLLM — Building the fastest AI gateway in Rust.
", "url": "https://development-wec.wiline.com/docs/news/litellm-rust-gateway/", "title": "Why LiteLLM Is Rewriting Its Gateway in Rust — and Why AI Developers Should Care", "summary": "LiteLLM is moving its AI gateway from Python to Rust. It's a signal that AI gateways are becoming critical infrastructure — with real consequences for latency, cost, and reliability on WiLine Edge Cloud.", "date_modified": "2026-06-24T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai-news", "gateways", "performance", "self-hosting" ] } ] }