A reader of the Cybersecurity Architect's Handbook Second Edition recently asked whether the Kubernetes sidecar pattern discussed in Chapter 13 applies only when an organization hosts its own large language model. The short answer is no..
The sidecar is an inference traffic control. It can inspect prompts sent to a model and responses returned to an application. That function remains useful whether the model runs in the same cluster, in another internal environment, or behind a vendor's API.
The distinction matters because request-path controls and model-internal controls solve different problems. A sidecar, gateway, or application library can inspect the data your application exchanges with a model. It cannot give you direct control over a vendor's model weights, training pipeline, dataset provenance, or registry. Those controls belong to the model operator and must be addressed through provider assurance, contracts, technical configuration, and the shared responsibility model.
What Chapter 13 describes
Pages 440 through 447 frame the Kubernetes sidecar as one of three places to enforce AI firewall and guardrail policy:
- A centralized API gateway that routes requests to internal models or external providers
- A sidecar proxy attached to a workload in Kubernetes
- An SDK or in-process library built into the application
The gateway pattern explicitly includes external providers. Every inference request passes through the gateway, where policy can be applied to the request and again to the response.
The sidecar pattern is more localized. It intercepts inbound and outbound traffic for a pod that might contain a model server, an agent runtime, or an application that calls AI APIs. That last case is the vendor-hosted scenario. The application does not need to host the model for the sidecar to inspect its AI traffic.
It is easy to miss this because a sidecar can sit beside a local model server. That is one use case, but it is not the boundary of the pattern.
What the control can do with a vendor LLM
When an application calls a vendor API, the security problem does not disappear. The trust boundary moves.
The outbound request may contain a system prompt, user content, retrieved documents, source code, credentials, personal data, regulated data, or instructions for an autonomous agent. The response may contain unsafe content, fabricated facts, malicious markup, leaked prompt material, or instructions that cause the application to call a tool. An AI firewall can inspect both directions before the application or provider receives the data.
Depending on the product and policy design, that inspection point can:
- Detect prompt injection and jailbreak patterns
- Identify secrets, personal data, or prohibited data before it leaves the environment
- Enforce model and provider allowlists
- Validate request size, model parameters, and permitted API operations
- Screen responses for unsafe content or sensitive-data disclosure
- Apply rate limits, quotas, and abuse controls
- Record security metadata for incident response and monitoring
A sidecar does not gain special access to the vendor's model. It governs the exchange between your workload and that vendor.
There is an important implementation condition: the traffic must actually pass through the sidecar in a form it can inspect. Most vendor API traffic uses TLS. A transparent network sidecar cannot read encrypted prompts and responses merely because it shares the pod's network namespace. The application must use the sidecar as an explicit proxy, connect through a service-mesh or egress design that terminates and re-establishes TLS, or emit the relevant content through an application-aware integration. Certificate trust, hostname validation, streaming responses, connection reuse, retries, and provider authentication all have to be handled correctly.
If that routing or encryption design is incomplete, the sidecar may see only destinations and connection metadata. That is useful for egress control, but it is not prompt and response inspection.
Gateway, sidecar, or SDK
All three patterns can work with a vendor-hosted model. The better question is which placement gives the organization the required coverage without creating an unreasonable operational burden.
Centralized gateway
A gateway is usually the cleanest enterprise control point. Applications call the gateway, and the gateway calls approved providers. This provides one place for policy enforcement, provider routing, authentication, rate limiting, and security telemetry.
The gateway sees the serialized request and response. It can therefore inspect prompts, completions, model identifiers, token usage, latency, response codes, and other protocol metadata. It also gives security teams a consistent point from which to feed events into a SIEM.
The tradeoff is concentration. If every AI request depends on one gateway service, its availability and policy health become part of the availability of every dependent application.
Kubernetes sidecar
A sidecar places the enforcement point beside the application. It can provide workload-specific policy and reduce dependence on a distant shared gateway. It is useful when a workload cannot use the enterprise gateway, needs local enforcement, or requires a separate trust boundary.
The cost is operational scale. Every protected workload now carries another component that must be configured, patched, observed, and kept consistent. Sidecars also consume compute resources and can complicate connection handling, certificate management, debugging, and application startup.
A sidecar is not automatically safer than a gateway. Its value depends on complete traffic capture, resistant bypass controls, trustworthy policy distribution, and clear behavior when the sidecar or its dependencies fail.
SDK or in-process library
An SDK has the best view of application intent. It can see the original system prompt, retrieved documents, tool selections, agent state, and other context that may never appear as distinct fields on the wire.
That fidelity comes with tighter coupling. Each application team has to integrate and maintain the library. Language support, version drift, inconsistent implementation, and developer bypass are common concerns. A library failure can also become an application failure.
For many enterprises, the practical design is a combination. A gateway provides common provider access, coarse policy, and centralized telemetry. SDK instrumentation adds internal application context where the risk justifies it. Sidecars cover workloads that need local enforcement or cannot use the normal gateway path.
The gateway single point of failure problem
A centralized AI gateway can become a single point of failure, but adding replicas solves only the simplest version of that problem.
The first concern is ordinary service availability. The baseline is multiple stateless gateway instances behind a load balancer across separate availability zones. Session and policy state should not be tied to one gateway instance. Health checks must exercise the inspection path rather than confirm only that the process accepts TCP connections.
If the policy engine or scoring service is unavailable while the gateway still reports healthy, traffic may pass without inspection. That is a security failure disguised as availability.
The next concern is degraded-state behavior. The architecture must define whether traffic fails open or fails closed when inspection is unavailable.
Failing closed preserves enforcement but makes the security control part of the application's availability chain. Failing open preserves access but creates a period in which prompts, responses, and agent actions may bypass inspection. There is no sound enterprise-wide default for every flow.
I prefer to tie this decision to data classification and consequence. An agent that can modify production systems, move money, approve access, or process regulated data should normally fail closed. A low-risk assistant working with public information may be allowed to fail open for a limited period, provided the bypass is logged, alerted, and explicitly accepted by the risk owner.
The third concern is correlated failure. Multiple gateway replicas still fail together when they depend on one policy store, one classifier, one identity service, or one global configuration push. Policy should be replicated and cached locally. Scoring services need their own redundancy. Configuration changes should be staged, canaried, and easy to roll back.
For large environments, I prefer independent gateway cells by region or business unit over one global gateway tier. A cell limits the failure domain. A gateway and sidecar combination can provide another enforcement layer, but only if the two controls have defined responsibilities and independently available policy. Duplicating the same dependency in both places does not create meaningful resilience.
The first design decisions I would document are:
- The fail-open or fail-closed behavior for each data and action class
- The gateway recovery objectives, regional posture, and dependency model
- The policy distribution, local caching, canary, and rollback process
- The approved bypass paths and the controls that prevent applications from silently using them
These decisions connect to NIST SP 800-53 controls such as SC-5 for denial-of-service protection, SC-6 for resource availability, and CP-10 for system recovery and reconstitution. If the service carries a formal availability commitment, the same design will also matter to a SOC 2 Availability assessment.
Session logging is not limited to sidecars and SDKs
A gateway can capture AI session context. In many environments, it is the best place to collect a consistent enterprise record because it sees requests and responses from many applications.
The three placements differ in what they know.
A gateway records what crosses the centralized provider boundary. A sidecar records what crosses the workload boundary. Both can capture the prompt and completion when they can inspect the application protocol. An SDK can capture application state that may not appear separately in the request, such as the document selected by retrieval, the tool the agent considered, the unrendered prompt template, or the reason a workflow chose a particular action.
That does not mean every prompt and response should be stored in full. Central logging can create a high-value collection of secrets, personal data, regulated records, proprietary documents, and authentication material. Logging policy should define field-level redaction, encryption, access control, retention, regional storage, legal holds, and incident-response use. In some cases, the right record is a policy decision, a content classification, a rule identifier, and a cryptographic reference to separately protected evidence rather than a raw prompt.
NIST SP 800-53 AU-2, AU-3, and AU-12 provide a useful foundation for deciding which events to record, what each record must contain, and how audit records are generated. The AI-specific work is identifying the context needed to reconstruct a prompt injection attempt, prohibited disclosure, model-extraction pattern, or unsafe agent action without turning the logging platform into a second uncontrolled data repository.
The architecture decision
Vendor-hosted LLMs do not remove the need for request and response controls. They make the outbound trust boundary more important.
A centralized gateway is often the best default because it gives the organization a consistent provider boundary and a central logging point. Sidecars make sense when enforcement needs to stay close to a Kubernetes workload, when a workload cannot use the shared gateway, or when local policy must remain available through a gateway failure. SDK controls belong where the application has security-relevant context that a proxy cannot infer.
The final design may use all three, but each component needs a specific job. If the gateway, sidecar, and SDK all apply overlapping policy without a clear source of truth, the result will be inconsistent decisions, difficult troubleshooting, and more ways to fail.
The sidecar pattern in Chapter 13 is not limited to self-hosted models. It applies to applications that call vendor APIs as long as the architecture routes inspectable inference traffic through the sidecar. The model's location changes which controls you own. It does not eliminate the need to govern the data and actions that cross your boundary.
Source note - This article expands on the AI firewall and guardrail placement patterns discussed in Chapter 13, of Cybersecurity Architect's Handbook, Second Edition.
