technical-term · Models & inference · foundational · Reviewed
Self-hosted inference
An inference service whose runtime, capacity, configuration, model artifacts, and operational controls are managed by the organization using it or by a directly governed operator.
Definition
Self-hosted inference places responsibility for deploying and operating the model-serving stack with the organization rather than consuming only a vendor-managed API. It may run on premises, in a private cloud, or in controlled hosted infrastructure. Self-hosting provides configuration and data-path options, but also transfers responsibility for scaling, patching, authentication, isolation, observability, evaluation, and incident response.
Model-serving software is established, while efficient and secure operation of rapidly changing models remains a specialized infrastructure discipline.
Why it matters
A self-hosted endpoint can serve many applications and therefore may be remote from each caller even though the organization controls the service.
The serving runtime is one layer. Gateways, identity, policy, quotas, secrets, data retention, model governance, and application-level verification remain separate responsibilities.
System anatomy
- Serving runtime
- Model loading, scheduling, batching, generation, and response handling.
- Controlled infrastructure
- Compute, network, storage, and administrative boundary operated under defined responsibility.
- Service controls
- Authentication, quotas, isolation, configuration, patching, and availability.
- Model operations
- Artifact approval, evaluation, rollout, rollback, and retirement.
Important distinctions
- Local inference
- A self-hosted service may run remotely inside organizational infrastructure and serve many clients.
- Managed private endpoint
- A vendor-operated dedicated endpoint can offer isolation without transferring full operational control to the customer.
Implementation signals
- Define service ownership and patch responsibilities
- Authenticate clients and isolate tenants before exposing the endpoint
- Version and canary model, runtime, and configuration changes
- Capacity-test realistic concurrent workloads
Failure modes
- An unauthenticated compatible API is exposed on the network
- A model update changes behavior without acceptance evaluation
- Operators underestimate GPU capacity, queueing, or rollback needs
Sources and further study
- vLLM — OpenAI-Compatible Server
Documents serving supported models through an HTTP API, including runtime arguments, model configuration, parallelism, request options, and deployment-oriented server behavior.
Use in this library: Official guidance. An inference server is not an identity, authorization, control, evaluation, or evidence layer. Defaults and compatibility vary by model, runtime version, and deployment.
guidance · guidance · Published 2026 - ggml.org — llama.cpp
Documents a C and C++ runtime for large-model inference across a wide range of local and cloud hardware, including GGUF models, quantization support, and an API server.
Use in this library: First-party case study. This is implementation documentation for an inference runtime, not a complete agent, security boundary, governance system, or guarantee of model quality and performance.
engineering · engineering · Published 2026 - Ollama — Ollama FAQ
Documents local prompt handling, local-only mode, networking, storage, and runtime configuration for operating models through Ollama on user-controlled hardware.
Use in this library: Official guidance. Vendor documentation describes runtime behavior and configuration. Local execution alone does not secure the host, logs, network, tools, model license, or governance process.
guidance · guidance · Published 2026