Author: Swati Sehgal (Red Hat)
The Device Plugin framework was introduced in the Kubernetes v1.8 release as a vendor independent framework to enable discovery, advertisement and allocation of external devices without modifying core Kubernetes. The feature graduated to Beta in v1.10. With the recent release of Kubernetes v1.26, Device Manager is now generally available (GA).
Within the kubelet, the Device Manager facilitates communication with device plugins using gRPC through Unix sockets. Device Manager and Device plugins both act as gRPC servers and clients by serving and connecting to the exposed gRPC services respectively. Device plugins serve a gRPC service that kubelet connects to for device discovery, advertisement (as extended resources) and allocation. Device Manager connects to the
Registration gRPC service served by kubelet to register itself with kubelet.
Please refer to the documentation for an example on how a pod can request a device exposed to the cluster by a device plugin.
Here are some example implementations of device plugins:
- AMD GPU device plugin
- Collection of Intel device plugins for Kubernetes
- NVIDIA device plugin for Kubernetes
- SRIOV network device plugin for Kubernetes
deviceplugin API packages moved from
k8s.io/kubelet/pkg/apis/ in v1.17. Refer to Move external facing kubelet apis to staging for more details on the rationale behind this change.
Additional gRPC endpoints introduced:
GetDevicePluginOptionsis used by device plugins to communicate options to the
DeviceManagerin order to indicate if
GetPreferredAllocationor other future optional calls are supported and can be called before making devices available to the container.
GetPreferredAllocationallows a device plugin to forward allocation preferrence to the
DeviceManagerso it can incorporate this information into its allocation decisions. The
DeviceManagerwill call out to a plugin at pod admission time asking for a preferred device allocation of a given size from a list of available devices to make a more informed decision. E.g. Specifying inter-device constraints to indicate preferrence on best-connected set of devices when allocating devices to a container.
PreStartContaineris called before each container start if indicated by device plugins during registration phase. It allows Device Plugins to run device specific operations on the Devices requested. E.g. reconfiguring or reprogramming FPGAs before the container starts running.
Pull Requests that introduced these changes are here:
- Invoke preStart RPC call before container start, if desired by plugin
- Add GetPreferredAllocation() call to the v1beta1 device plugin API
With introduction of the above endpoints the interaction between Device Manager in kubelet and Device Manager can be shown as below:
Device plugin code was refactored to separate 'plugin' package under the
devicemanagerpackage to lay the groundwork for introducing a
v1beta2 device plugin API. This would allow adding support in
devicemanager to service multiple device plugin APIs at the same time.
With this refactoring work, it is now mandatory for a device plugin to start serving its gRPC service before registering itself with kubelet. Previously, these two operations were asynchronous and device plugin could register itself before starting its gRPC server which is no longer the case. For more details, refer to PR #109016 and Issue #112395.
In Kubernetes 1.26, inspired by how Persistent Volumesare handled in Kubernetes, Dynamic Resource Allocationhas been introduced to cater to devices that have more sophisticated resource requirements like:
- Decouple device initialization and allocation from the pod lifecycle.
- Facilitate dynamic sharing of devices between containers and pods.
- Support custom resource-specific parameters
- Enable resource-specific setup and cleanup actions
- Enable support for Network-attached resources, not just node-local resources
No, the Device Plugin API is still not stable; the latest Device Plugin API version available is
v1beta1. There are plans in the community to introduce
v1beta2 API to service multiple plugin APIs at once. A per-API call with request/response types would allow adding support for newer API versions without explicitly bumping the API.
In addition to that, there are existing proposals in the community to introduce additional endpoints KEP-3162: Add Deallocate and PostStopContainer to Device Manager API.