Oracle Details Kubernetes Management for Its Private LLM Service

Oracle has published a Kubernetes deployment walkthrough for its Private AI Services Container 26.2, showing how OraOperator 2.2 can download a Hugging Face model and run it on a GPU-enabled node. The setup extends Oracle’s Private Large Language Model Service, announced in August, for organizations that want open-weight models inside infrastructure they control. Customers provide and operate the hardware.

The new guidance is a practical layer on top of the service release. Oracle says OraOperator manages certificates, secrets, storage for downloaded models, configuration, networking, scaling, and placement in a Kubernetes cluster. Its example uses Oracle Kubernetes Engine and calls for at least one GPU node, such as one with an NVIDIA A10 GPU.

From a model repository to a local endpoint

The container can download and run compatible models from Hugging Face repositories. Some models require a Hugging Face token and license acceptance. Oracle’s example configures OpenAI’s open-weight gpt-oss-20b with the vLLM runtime, a 16,384-token maximum model length, and a GPU-enabled Kubernetes node.

Three execution choices

• Oracle-adapted ONNX runtime handles embedding workloads on CPUs.
• llama.cpp runs the described CPU-oriented GGUF model format.
• vLLM supports NVIDIA GPUs and CPUs for model inference.

Compatibility removes one integration hurdle, not the operating work

The service implements the OpenAI API interface, which Oracle says lets compatible clients connect by changing an endpoint URL and API key. It also supports embeddings, vector indexes, and text-generation capabilities through OpenAI-compatible inference APIs. The large container includes an optional Ministral-3-3B-Reasoning-2512 model, alongside the option to fetch other compatible open-weight models.

Oracle’s documentation draws a practical line between the runtimes: llama.cpp is suited to exploration, while vLLM is generally preferred for enterprise deployments because it handles concurrent requests more efficiently. Each vLLM- or llama.cpp-backed model runs in its own server, with one such model supported per container instance. Self-hosting can move model traffic into a customer-controlled environment, but it also makes capacity, placement, and model operations part of the deployment decision.