Platform / Inference & Model Governance
Four decisions, in order: which models may be called at all, what may be sent, what may come back, and what may then happen. Same policy, same approval path, same evidence chain across all four — reached by the LLM gateway you already run, not by a proxy inserted in front of your model provider.
loading…The caller keeps the connection and the credentials. GovernorAI returns a verdict.
TOPOLOGY
Most controls on inference arrive as a proxy: you re-point your applications at a new hostname, hand over the provider keys, and take on a new dependency in the availability path of every model call. GovernorAI does the opposite. It registers as a guardrail on the gateway that is already there — a Portkey webhook check, a LiteLLM CustomGuardrail, an Envoy ext_proc filter — and answers a question. The gateway keeps the provider credentials, the connection and the retry behaviour.
| Adapter | Attaches to | On allow | On deny, pause or error |
|---|---|---|---|
| Portkey webhook | POST /v1/portkey/guardrail — a Bring Your Own Guardrail check on the before- and after-request hooks. Config only; no code on the Portkey side. | verdict:true, with a redacted body returned as transformedData when Portkey sends the structured JSON. | HTTP 200 with verdict:false — never a 5xx. |
| LiteLLM guardrail | POST /v1/llm/check through governor_guardrail.py, wired at pre_call, during_call and post_call. | allow:true, with transformed_messages or transformed_response when content was shaped. | allow:false carrying fail_closed. The shim blocks on an outage unless fail_open is deliberately set. |
Envoy ext_proc | Envoy Gateway, Istio, Gloo, Envoy AI Gateway, agentgateway — gRPC on :9002. | CONTINUE; a redaction is applied as a body mutation. | Pause returns 202, deny returns 403. failure_mode_allow: false is required. |
| Forward-auth | NGINX auth_request, Traefik, Apigee, Azure APIM, Lambda authorizer. | 200. | 403. This pattern cannot rewrite a body — it emits X-Governor-Shaping-Required: true so the operator knows shaping was requested but not applied. |
Portkey's webhook guardrail times out at three seconds and, on a timeout or transport error, defaults to verdict:true. A shim that returned a 5xx on its own internal error would therefore let the model call through. So the adapter never returns a 5xx: a genuine deny, a held approval, an unparseable body and an internal fault are all surfaced in-band as HTTP 200 with verdict:false, and each decision is bounded to a sub-three-second deadline (2500ms by default) so the answer lands before the gateway's timeout fires. The operator must still set the guardrail's failure action to deny — GovernorAI cannot set that from its side, and the deployment guide says so rather than assuming it.
This is data control and injection control on inference, plus control over which models may be called. It is not model output quality, model evaluation, or any judgement about whether an answer is correct: GovernorAI does not score a completion, rank a model, or assess an answer's truthfulness. What it does do is decide, per call, whether the model may be called, whether the prompt may be sent, whether the response may be returned, and whether the action that follows may proceed — and leave an attributable record of each of those four decisions.
MODEL → REQUEST → RESPONSE → ACTION
A prompt that leaks a customer identifier, a completion that carries an instruction planted in a retrieved document, and a tool call that writes to a system of record are not three products. They are four positions on the same call, and the decision at each of them is taken by the same decision core against the same policy.
Every model is a registry row with a status: pending review, approved, restricted, deprecated or blocked. A model policy carries allowed and blocked model lists, a maximum risk tier and a tier above which approval is required. Approving a model writes who approved it and when.
Models discovered by Shadow AI can be promoted into the registry, and they land as pending review — governed, but not yet approved.
status · risk_tier · approved_by · approved_atThe prompt is a governed surface, not a log line. Policy is authored against llm.chat.request, whose arguments carry the messages and the model name. Detectors run over the content: secrets are redacted, sensitive data and regulated identifiers are masked, and prompt injection is a deny-class signal because it cannot be safely rewritten.
The completion is inspected on the way back, against llm.chat.response. Where the response is the output of a governed retrieval call, an operator can declare it untrusted retrieved content, and the indirect-injection detector inspects it as a payload rather than as an answer — never inferred from a tool name, only from operator configuration or an explicit declaration.
What the model said becomes a tool call, and that is the action seam. The same core decides it: allow, deny, hold for approval, or narrow the arguments and re-verify them against the exact payload before dispatch. The verdict, its rule and its evidence entry are written to the same hash-chained record as the three decisions before it.
POST /api/v1/gateway/executeModels are approved, restricted or blocked at the seam — the registry holds the governance state, the reviewer and the timestamp, and the decision at the call is what actually stops it. Every usage record carries the decision that was reached, and the agent-to-model link is a durable declared association rather than an inference from recent traffic, so a quiet agent does not silently lose its edge in the governance graph.
SEAM CAPABILITY
Allow, deny and pause are universal. The richer outcomes — redact, mask, constrain — are not, and a seam that cannot execute an outcome is not permitted to advertise it. The capability matrix is a registry in the runtime, not a marketing table: when a policy resolves to an outcome the seam cannot perform, the decision is downgraded to deny and marked as downgraded, with the original outcome retained in the decision detail.
| Seam | Request side | Response side | Stated limit |
|---|---|---|---|
gateway_execute | Redact, mask, constrain | Redact and mask, including plain-text and SSE bodies shaped in flight | Constraining is request-side only. A response that cannot be shaped safely fails closed to deny rather than being forwarded un-inspected. |
mcp_invocation | Redact, mask, constrain — argument-aware | Retrieval output inspected as untrusted retrieved content | Ordinary model-answer responses are not classified on this seam. |
provider_bedrock | Redact, mask, constrain — verified against the exact payload sent onward | Redact and mask on the response text, with a re-scan that proves the content is gone | Streaming, SSE and binary Bedrock responses are not shaped. Content a detector cannot localize fails closed rather than being partially redacted. |
provider_azure | Constrain, re-verified against the exact request | Not offered | Response shaping is not available on this seam. A constraint that cannot be honoured fails closed to deny. |
provider_gcp | Constrain, re-verified against the exact request | Not offered | Response shaping is not available on this seam. |
sdk_wrapper | Consult only — allow, deny, pause | Not offered | The SDK path receives a decision, not a shaped payload; choosing a shaping outcome here downgrades to deny. |
Each seam declares the interaction kinds it can actually inspect and the decisions it can actually carry out. The inspector consults that declaration before applying a richer outcome, and an unregistered seam falls back to allow, deny and pause with no shaping claimed at all. Adding an outcome to the matrix requires shipping the runtime support first.
You cannot un-send a streamed chunk. On a streaming response the shaper redacts and masks within a bounded sliding window, and it terminates the stream — withholding the remainder — when it meets a deny-class signal such as prompt injection, or a finding it cannot localize precisely enough to remove. Bytes already delivered to the client are gone. For the same reason LiteLLM's post-call streaming hook is audit-only: blocking on streaming traffic has to be enforced on the input, at the pre-call or moderation stage, and the deployment guide says that plainly rather than implying the output hook can stop it.
DEPLOYMENT
Nothing about the model call moves. The provider key stays where it is, the model list stays where it is, and GovernorAI is reachable at an address the gateway calls. The three hooks below are the same decision core answering at three moments of the same call.
# the model list is untouched — your key stays yours
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
guardrails:
# block / redact the INPUT before the model call.
# the enforcement seam, and the only one that
# works for streaming responses.
- guardrail_name: "governorai-pre"
litellm_params:
guardrail: governor_guardrail.GovernorAIGuardrail
mode: "pre_call"
api_base: os.environ/GOVERNOR_LLMGUARD_URL
fail_open: false # fail CLOSED on outage
# govern a fully buffered response. streaming
# responses are audit-only post-call.
- guardrail_name: "governorai-post"
litellm_params:
guardrail: governor_guardrail.GovernorAIGuardrail
mode: "post_call"
api_base: os.environ/GOVERNOR_LLMGUARD_URL
fail_open: false# allowed, with the prompt rewritten by inline DLP
HTTP 200
{
"verdict": true,
"transformedData": { "request": { … } }
}
# denied. still HTTP 200 — a 5xx would trip
# Portkey's fail-OPEN default and let the
# model call through.
HTTP 200
{ "verdict": false }
# a pause is also verdict:false — a synchronous
# webhook cannot hold a request open while a
# human decides. the approval is recorded and
# the call is refused now.
# operator action required on the Portkey side:
# guardrail failure action = deny
# tenancy travels on the webhook call, so the
# verdict is attributable to a principal:
# x-governor-org-id
# x-governor-principal-id
# x-governor-agent-idBecause the gateway consults rather than forwards, the provider credentials, the connection and the retry behaviour stay with the gateway team. What GovernorAI adds is the verdict and the record. Where the boundary itself is contested — an underwriting model, a named model id, a whole decision domain — a tenant can declare those surfaces out of scope and generate an attestation over the runtime evidence showing that no governance decision touched any of them.