What happens to your prompt before it reaches the model?
In an organization, no tool should call a model directly. Every request goes through a single gateway that enforces policies. Engineer view: at each step, the policy and its configuration snippet, the return code and the OWASP risk it addresses.
In an organization, no tool should call a model directly. Every request goes through a single gateway that checks identity, protects client data and keeps costs under control. Run a scenario and follow the request.
- 5controls
- 4scenarios
- 6verified sources
How a request travels
3 key takeaways
One single point of passage
Tools never call a model directly. Controls are applied once, in one place, for everyone.
Sensitive data stays protected
Masking before anything leaves, and confidential requests routed to an on-premises model.
Every token has an owner
Per-team quotas, token counting and project chargeback make AI costs visible and manageable.
The steps
Good to know
Reading the diagram
- Security
- Costs
- Traceability
- Request
- Response
- Blocked request
The “Source” labels link to the document that describes the control. The dashed “Adaptation” label marks a rule I propose for an organization, which does not come from a source. Figures and names in the examples are fictitious. Policy snippets follow the Microsoft documentation syntax with fictitious values; the mapping to OWASP risks is a proposal.
What this architecture does not guarantee
- It reduces the risks of data leaks and cost drift; it does not remove them. Usage rules and training are still needed.
- A cloud-hosted gateway adds latency for on-premises services. A self-hosted gateway avoids that trip, but the company must deploy, size and maintain it.
- Logging content has limits: the size of captured messages, and sampling, which can also weigh on throughput.
- Metrics are not an audit log: Microsoft states that Application Insights is not designed as an audit system. An audit log needs dedicated storage, with a set retention period.
Sources
- Microsoft, GenAI gateway reference architecture using Azure API Management : managed-identity authentication, personal-data masking, tokens-per-minute limits, spillover from reserved capacity to pay as you go, token counters for chargeback, self-hosted gateway.
- API7, What is an AI gateway : provider keys held by the gateway, per-team budgets, routing by cost and task, audit logs.
- OWASP, LLM10:2025 Unbounded Consumption : uncontrolled use of a model, up to unexpected costs (“denial of wallet”).
- Microsoft Learn, API Management AI gateway policies : llm-token-limit, llm-content-safety, llm-semantic-cache-lookup, llm-emit-token-metric (syntax, 403 and 429 codes, warnings).
- Microsoft Learn, validate-jwt and authentication-managed-identity : token validation (401 by default) and gateway authentication to the provider.
- OpenTelemetry, generative AI observability : gen_ai.client.token.usage and gen_ai.client.operation.duration metrics; conventions still under active development.
Microsoft documents an implementation on Azure. The principle of a central gateway applies to other products.