Oracle adds Kubernetes-controlled repairs for faulty GPU hosts

A GPU host can look ready in Kubernetes while its hardware is failing. Oracle announced OKE GPU Host Repair on October 26, 2026, bringing fault reporting and coordinated recovery into Kubernetes. It can arrange reboots or instance termination for host repair, while customers set limits on how much running work those actions can disrupt.

A ready node can hide a hardware fault

GPU errors, degraded RDMA links—the network connections between hosts—and physical host problems can reduce performance or prevent workloads from running. Kubernetes may still mark the worker node ready.

The GPU Health Monitoring add-on exposes supported GPU and RDMA signals as Kubernetes node conditions. A per-node NodeHealthReport records faults, recommended actions, maintenance information and repair progress. Customers select which nodes to monitor.

Three triggers, different recovery actions

An OKE-managed controller watches those signals. For qualifying repairs, it blocks new workloads and attempts to evict existing ones before coordinating the applicable OCI Compute action. Three recovery paths determine what happens next:

• Operator-triggered repair: An authorized operator labels a node based on customer-identified problems, even without an eligible automated signal. OKE attempts eviction and terminates the instance. Live use requires tenancy enablement through Oracle Support.
• GPU instance recovery: Persistent, qualifying signals recommending REBOOT can trigger an automated reboot. A cooldown limits repeated attempts. CUSTOMER_ACTION_REQUIRED signals remain visible but do not trigger automatic reboots.
• Maintenance-event repair: Compute events specify termination for host repair or stopping the instance. For a STOP action, OKE tracks repair completion and returns the node to a state where it can accept workloads.

Customers decide how disruptive repair can be

Long-running GPU jobs, large saved checkpoints and limited spare capacity complicate repair decisions. NodeRepairConfig lets teams select covered nodes, cap parallel repairs and halt new repairs when too many nodes are unhealthy. Customers also choose how long eviction can take and whether repair should proceed if eviction fails.

Draining respects Kubernetes eviction controls, including pod disruption budgets. Dry-run mode records a proposed workflow without blocking workloads, evicting them or requesting a reboot or termination. An optional approval pause leaves the node able to accept workloads while customers investigate, then approve or cancel repair.

The feature supports eligible managed GPU worker nodes in enhanced OKE clusters running Kubernetes 1.32 or later, subject to supported instance types and regional availability.