Back to the CIO and CSO starting point Security and GRC hub
AI Hosting Options: Private Endpoints, Self-Hosting and Residency
AI hosting options fall into two groups: hosted routes with private connectivity, and self-hosting. Lightbridge advises CIOs and CSOs to treat them as different transfers of risk. A private endpoint narrows network exposure. Customer-managed keys and residency settings each cover specific data only. Self-hosting moves every control, and every failure, to your own team.
This page sets out what each route does and does not guarantee, compares private connectivity, customer-managed keys and residency across three clouds, lists what self-hosting adds, and gives a sequence for deciding. Vendor statements are attributed and linked. Each source carries the date its page shows, and pages with no date are marked undated. Lightbridge does not recommend a vendor here. It is part of Lightbridge's security and GRC advisory hub.
What Lightbridge means by a hosted route and by self-hosting.
A hosted route sends your prompts to a model that a cloud provider runs. Private connectivity changes how traffic reaches it. It does not change who operates the model. Self-hosting means your team runs the model server, the network around it, and the model files. Lightbridge's AI access paths page compares what each vendor route does with your data once it arrives. This page covers the layer around that: the network path, the keys, the location, and the operator.
Three facts frame the decision. On AWS, the provider states that model providers have no access to the Bedrock Model Deployment Accounts, and so no access to Bedrock logs or customer prompts and completions (AWS). Private connectivity scope varies by feature on every cloud we reviewed. And no formal responsibility split between hosted and self-hosted routes was in the primary sources we fetched. That last point is a gap, and Lightbridge treats it as one.
Private connectivity, keys and residency on each cloud, as Lightbridge reads the documentation.
Each cell says what the vendor states. Where we found no primary statement, the cell says so, and where we did not review a topic, the cell says that instead. Source links carry the page's own date where it shows one. The AWS, vLLM and Ollama pages and most Google pages show none, so they are undated here.
| Route | Private connectivity | Customer-managed keys | Residency and its limits |
|---|---|---|---|
| AWS Bedrock | AWS states that interface VPC endpoints (PrivateLink) exist for separate service names, including bedrock-runtime (inference API), bedrock (control plane), bedrock-agent, bedrock-agent-runtime and bedrock-mantle, plus FIPS variants (source). AWS states a VPC with an interface endpoint can keep data from being available over the internet, and lists model customization, batch inference and Knowledge Bases (source). The page does not state that this covers third-party SaaS or a model provider's own endpoints. Whether PrivateLink covers cross-Region routing end to end: Not stated in the documentation we reviewed. | AWS lists the resources a customer KMS key can encrypt: model customization jobs and output custom models, agents, knowledge base ingestion jobs, OpenSearch vector stores, and model evaluation jobs, plus S3 data via SSE-KMS (source). On-demand inference prompts and responses are not in that list. An explicit AWS statement on inference-time prompt encryption with customer keys: Not stated in the documentation we reviewed. | AWS states that cross-Region inference selects a commercial Region within the chosen geography, or any commercial Region for a global profile. It states data stays on the AWS network, is encrypted in transit between Regions, and CloudTrail records the processing Region in the source Region (source). |
| Google Vertex AI (Gemini Enterprise Agent Platform) | Google states VPC Service Controls creates a perimeter that keeps specified artifacts, including online inference requests and Gemini models, from leaving it, and that public internet access is blocked once a perimeter protects the service unless the source is allowlisted. Limits it states: URL context live fetching is disabled, request and response logging is unavailable, Model Garden 1-click public deployments are unsupported, and endpoints must be created after the project joins the perimeter (source). Private services access endpoints support one model per endpoint, do not support SSL/TLS, and do not support tuned foundation models (source). Private Service Connect for Gemini inference: Not reviewed for this page. | Google states CMEK does not encrypt operation metadata, that for endpoints it covers model files but not in-memory data, and that it is unsupported for TPU tuning, AutoML image batch prediction, and Memory Bank or Sessions using the global endpoint (source). | Google states global endpoints may process requests in any Google Cloud location and give no residency guarantee. Jurisdictional multi-region endpoints keep ML processing within the stated region such as US or EU. Locational endpoints keep it within the country or multi-region jurisdiction. Data at rest stays in the chosen location (Google, updated 2026-10-30). Google adds that multi-region endpoints are foundational toward DoD IL5 or ITAR, and full compliance needs broader customer-managed architecture, identity, and access controls. |
| Azure OpenAI and Microsoft Foundry | Microsoft states private endpoints route client traffic over a private link on the Azure backbone, let the firewall block the public endpoint, and can serve on-premises networks through VPN Gateway or ExpressRoute private peering. Clients must call the resource's custom subdomain. Azure OpenAI uses a different private DNS zone than other Foundry Tools. Name resolution from outside the virtual network resolves to the public endpoint (Microsoft, ms.date 2026-02-17). | Microsoft states customers can create, rotate, disable and revoke key access. Only RSA and RSA-HSM keys of size 2048 are supported. Rotating the key does not re-encrypt data already in the resource. Fine-tuned models sit in Azure Storage in the same region, encrypted with Microsoft-managed or customer-managed keys (Microsoft classic documentation, ms.date 2025-11-26). Coverage of prompts and completions during inference: Not stated in the documentation we reviewed. | Microsoft states data at rest stays in the designated geography. Global deployment types may process inference in any Azure region. Data Zone types process only within the stated zone (US, EU, APAC). Standard and Regional Provisioned types process within the chosen geography and might process between regions inside it for operational purposes. Microsoft can add regions to the EU or APAC zone without prior notice, and the Developer type carries no residency guarantee (Microsoft, ms.date 2026-08-06). |
Customer-managed key coverage for non-first-party models on each cloud, such as partner models: Not reviewed for this page. Residency claims here rest on vendor documentation only. We did not consult EU law texts or standards catalogs for this page.
What self-hosting adds to your team's list, according to the sources Lightbridge reviewed.
Self-hosting removes a third party from the data path. It also removes the third party's controls. The sources below describe what an operator must apply. They are not a formal responsibility split, and we did not find one.
| Area | What the source states | What it means for the operator |
|---|---|---|
| Multi-node serving network | The vLLM project states that communications between nodes in a multi-node deployment are insecure by default and must be protected by an isolated network, covering PyTorch Distributed, KV cache transfer, and tensor, pipeline and data parallel traffic. It cites PyTorch's guidance that messages are sent unencrypted and connections are accepted from anywhere without checks (vLLM). | Lightbridge's reading: network isolation for the inference cluster is your design task, and a flat network is a finding. |
| Listening ports | vLLM states torch.distributed TCP initialization creates a TCPStore listening on all network interfaces by default, which can be outside vLLM's direct control, and recommends blocking all incoming connections except the API server port (vLLM, undated). Ollama's FAQ states it binds to 127.0.0.1 port 11434 by default, changed with the OLLAMA_HOST variable (Ollama). Whether Ollama has built-in authentication when exposed: Not reviewed for this page. | Host firewall rules and a deliberate bind address. Changing a default bind to reach a server from another machine is a security decision that needs an owner. |
| Server-side request forgery | vLLM states that without --allowed-media-domains a user could supply media URLs that reach internal services or cloud metadata endpoints, and documents download and decode size limits, such as a default of 256 MB for VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB, that it does not recommend disabling for deployments exposed to untrusted users (vLLM, undated). | Set the allowed domains and keep the limits. Treat any multimodal endpoint as an outbound-request surface. |
| Model file integrity | OWASP's LLM03:2025 states that pre-trained models are binary black boxes where static inspection offers little assurance, advises models from verifiable sources with third-party integrity checks such as signing and file hashes, and states vulnerable models can come from poisoned datasets or direct tampering and can contain backdoors that a repository's safety evaluations miss (OWASP, modified 2025-05-05). Guidance on storing and encrypting weights at rest: Not reviewed for this page. | A model intake step: approved sources, recorded hashes, and a named owner for each model in production. |
| Secure development practices | NIST SP 800-218A (July 2024) is an SSDF Community Profile adding secure development practices specific to AI model development, for producers of AI models, producers of AI systems that use them, and acquirers of those systems (NIST). The UK NCSC guidelines for secure AI system development (version 1.0, 27 November 2023) cover supply chain security in development and protecting infrastructure and models from compromise, threat or loss in deployment, for any AI-using system including those built on others' services (NCSC). CISA, NSA, FBI and partners released AI Data Security guidance on 22 May 2025 covering data security and integrity risks across the AI lifecycle (CISA). | Reference frameworks for your control design. We read their scope statements only, not the task-level text, so check them directly before you cite a specific control. |
Kubernetes and container hardening guidance for model serving: Not reviewed for this page.
The sequence Lightbridge advises for deciding between hosted and self-hosted.
The order matters. Early steps can rule a route out before anyone prices it. The sequence is Lightbridge's planning judgment, not a vendor or standards recommendation.
- Classify the data first. List the data classes the workload will send to a model. Start from the data, since it fixes what each later step must prove. Shadow use is part of this picture, and the shadow AI guide covers how to find it.
- Write the residency promise you have made. Pull the commitments from customer contracts and regulators. If one says processing stays in a named country, global endpoints and global deployment types are out for that workload, based on what Google and Microsoft state above.
- Match the exact model and feature to a private connectivity option. Coverage differs by feature. A private endpoint for the inference API does not tell you about tuned models, logging, or agents.
- Ask what the keys actually cover. Get the resource list in writing for the models you plan to use. Do not assume a customer-managed key reaches prompts.
- Check the observability you give up. Google states request and response logging is unavailable under VPC Service Controls. Decide whether you can meet your audit needs without it. The observability and audit logging page covers what to keep.
- Cost the self-hosting staffing honestly. Use the table above as the checklist: network isolation, bind addresses, request filtering, model intake, patching. If you cannot name the owners, the route is not ready.
- Decide per workload, and set a review date. Documentation changes. Microsoft states it can add regions to its EU or APAC data zone without prior notice. Put a recurring check on the architecture record, on a fixed schedule you set.
Agents and tool access add a further layer that hosting does not solve. See AI agent security and AI identity and developer access.
What Lightbridge suggests you ask, and what to keep as evidence.
- Which private connectivity option covers the model, region and feature we will use, and what does it leave out?
- Which resources does a customer-managed key cover, and does it reach prompts and responses?
- Which endpoint or deployment type applies, and in what locations can inference run?
- What logging or features become unavailable when the private configuration is on?
- For self-hosting: who owns network isolation, model intake, patching, and on-call?
Keep the answers with the date and the documentation link, plus an export of the endpoint and network configuration in force. Customers and auditors rarely accept a bare assertion of "private". The AI security questionnaires page shows how these answers feed the forms you will receive, and AI contracts and legal covers what belongs in the agreement.
How Lightbridge advises on AI hosting.
Lightbridge is an independent advisory and program-management firm. We help a CIO or CSO frame the hosting decision, gather the vendor answers, and run the program through delivery. Where hands-on configuration or operations are needed, vetted partners deliver under Lightbridge's program management. For the cloud infrastructure side of this decision, selection and architecture advisory across AWS, Azure and GCP sits with Lightbridge Cloud.
Lightbridge Automation, a sibling practice in the Lightbridge group, publishes explainers on zero data retention, LLM gateways, the NIST AI RMF, and the Claude compliance guide, and offers AI governance consulting. If a recommendation here points to work a sibling practice would deliver, Lightbridge discloses that upfront and you remain free to take the work elsewhere. For wider strategy, see Lightbridge's AI advisory.
AI hosting options: frequently asked questions answered by Lightbridge
Does a private endpoint mean our prompts never touch the internet?
Only for the traffic that uses the endpoint. AWS states that a VPC with a PrivateLink interface endpoint can keep data from being available over the internet, and lists model customization, batch inference and Knowledge Bases as covered features. Microsoft states that name resolution from outside the virtual network resolves to the public endpoint. Lightbridge advises testing resolution from every network that calls the model, not only the one you configured.
Do customer-managed keys encrypt our prompts and responses?
Check each platform before you assume so. AWS lists the Bedrock resources a customer KMS key can encrypt, and on-demand inference prompts and responses are not among them. Google states CMEK for endpoints covers model files but not in-memory data. The Microsoft customer-managed key page we reviewed (the classic-portal version) addresses stored data such as fine-tuned models, and does not say whether the keys cover prompts and completions during inference.
Can we guarantee data stays in one country or region?
Only with the right endpoint or deployment type. Google states that global endpoints may process requests in any Google Cloud location and give no residency guarantee. Microsoft states that Global deployment types may process inference in any Azure region. AWS states that cross-Region inference selects a commercial Region within the chosen geography, or any commercial Region for a global profile. Lightbridge advises recording the endpoint type in your architecture record, because the default may not match your promise to customers.
Does data residency make us compliant with ITAR or similar regimes?
Google states that multi-region endpoints are a foundational capability toward DoD IL5 or ITAR, and that full compliance requires broader customer-managed architecture, identity, and access controls. Lightbridge reads that as a general rule: residency settings are one input to a compliance case, never the whole case. Legal counsel and your auditor decide what your regime requires.
Is self-hosting safer than a hosted route?
It gives you control, and it hands you the work. The vLLM project states that communications between nodes in a multi-node deployment are insecure by default and must be protected by an isolated network. OWASP states that pre-trained models are binary black boxes where static inspection offers little assurance. We did not find a primary source that compares the two routes formally, so Lightbridge frames this as a staffing question: do you have the people to run it securely?
What should we ask a vendor about hosting?
Ask five things. Which private connectivity option covers the exact model and feature you will call. Which resources a customer-managed key covers. Which endpoint or deployment type applies, and where inference can run. What logging is unavailable under the private configuration. Who is responsible for each control when something fails. Record the answers with the date, since documentation changes.
Who decides between a hosted route and self-hosting?
Lightbridge recommends a joint decision by the CIO, the CSO, and the business owner of the data. Self-hosting has staffing and support costs that sit in IT, and exposure that sits in security. An independent advisory firm can run the comparison against your requirements, and partners deliver the build under Lightbridge program management. Lightbridge does not recommend a vendor on this page.
Settle the hosting question: talk to a Lightbridge advisor.
Bring your data classes, your residency commitments, and the routes under consideration. Lightbridge will help you test each against the vendor documentation.