2 posts tagged with "gateway"

Gateways · AI News

Pfizer runs an AI gateway — the single service every internal tool talks to when it wants a model. During a routine upgrade check, its throughput fell by half. No errors. No failed requests. Nothing wrong on any dashboard.

They published what they found together with the LiteLLM team, and the cause is small enough to fit in a paragraph: someone wrote a setting that said don't encrypt this connection, and the system encrypted it anyway.

Almost every "secure your LLM app" guide gives the same advice: before a prompt reaches the model, strip the personal data out of it — swap names, emails, and account numbers for [REDACTED] or <PERSON>, then call the model.

It sounds obviously right. I assumed it was, too. Then I went and read the research on what masking actually does to a model, and the papers point the other way. The short version: mask your logs, not your prompts.