A Storage Vulnerability Deep Dive

Kubernetes Safety is continually evolving – holding tempo with enhanced performance, usability and adaptability whereas additionally balancing the safety wants of a large and various set of use-cases.

Just lately, the GKE Safety workforce found a excessive severity vulnerability that allowed workloads to have entry to components of the host filesystem outdoors the mounted volumes boundaries. Though the vulnerability was patched again in September we thought it could be helpful to jot down up a extra in-depth evaluation of the difficulty to share with the neighborhood.

We assessed the affect of the vulnerability as described in vulnerability administration in open-source Kubernetes and labored carefully with the GKE Storage workforce and the Kubernetes Safety Response Committee to discover a repair. On this publish we’ll give some background on how the subpath storage system works, an summary of the vulnerability, the steps to seek out the basis trigger and the repair, and at last some suggestions for GKE and Anthos customers.

Kubernetes Filesystems: Intro to Quantity Subpath
The vulnerability, CVE-2021-25741, was attributable to a race situation through the creation of a subpath bind mount inside a container, and allowed an attacker to achieve unauthorized entry to the underlying node filesystem and its delicate information. We’ll describe how that system is meant to work, after which speak concerning the vulnerability.

The quantity subpath function in Kubernetes allows sharing a quantity in a number of containers inside a pod. For instance, we might create a Pod with an InitContainer that creates directories with pre-populated knowledge in a mounted filesystem quantity. These directories can then be utilized by containers in the identical Pod by mounting the identical quantity and optionally specifying a subpath subject to restrict what’s seen contained in the container.

Whereas there are some nice use instances for this function, it’s an space that has had vulnerabilities found up to now. The kubelet have to be additional cautious when dealing with user-owned subpaths as a result of it operates with privileges within the host. One vulnerability that has been beforehand found concerned the creation of a malicious workload the place an InitContainer would create a symlink pointing to any location within the host. For instance, the InitContainer might mount a quantity in /mnt and create a symlink /mnt/assault contained in the container pointing to /and so on. Later within the Pod lifecycle, one other container would try to mount the identical quantity with subpath assault. Whereas making ready the volumes for the container, the kubelet would find yourself following the symlink to the host’s /and so on as a substitute of the container’s /and so on, unknowingly exposing the host filesystem to the container. A earlier repair made positive that the subpath mount location is resolved and validated to level to a location inside the bottom quantity and that it isn’t changeable by the person in between the time the trail was validated and when the container runtime bind mounts it. This race situation is called time of examine to time of use (TOCTOU) the place the topic being validated adjustments after it has been validated.

These validations and others are summarized within the following container lifecycle sequence diagram.

Quantity subpath validations earlier than the container startup

A New TOCTOU Vulnerability: CVE-2021-25741
The newest vulnerability was found by performing a symlink assault much like the one defined above, with the distinction being that it consistently swapped the symlink with a listing in a decent loop, utilizing the RENAME_EXCHANGE possibility with renameat(2). If the timing is good, the kubelet will see the trail as a listing and go the validation examine. Then the mount utility might discover that the trail is a symlink pointing to the host and comply with it, exposing the host filesystem to the container. That is visualized within the following diagram:

The expectation and the assault final result

The GKE Safety and Storage groups labored carefully to revise the repair accomplished beforehand to discover a answer. The earlier repair takes a number of steps to make sure that the listing being mounted is safely opened and validated. After the file is opened and validated, the kubelet makes use of the magic-link path underneath /proc/[pid]/fd listing for all subsequent operations to make sure the file stays unchanged. Nonetheless, we discovered that all the efforts have been undone by the mount(8) linux utility which was dereferencing the procfs magic-link by default. As soon as the issue was understood, the repair concerned ensuring that the mount utility would not dereference the magic-links by utilizing the –no-canonicalize flag within the mount command.

The repair is in