This blog will discuss the need for better practices around the Pull Policy of a Kubernetes container, which is set using the imagePullPolicy field in a pod specification. When the imagePullPolicy is set to Always, users can ensure the latest version of the image is deployed every time a Kubernetes pod is started.
Relying on cached versions of a Docker image can introduce a security vulnerability because Kubernetes will attempt to use the cached version of an image without verifying where it came from. If an attacker is able to replace a cached image and the imagePullPolicy is set to IfNotPresent, Kubernetes will happily run any image you place on the node(s). Also, not using the Always pull policy can cause variations in images that are running per node.
For example, many projects will release version 1.2.3 with the image tags 1, 1.2, and 1.2.3. With regards to Kubernetes configuration, it is possible to specify tag 1.2, so when 1.2.4 is released with a security patch, it automatically flows in the next time that pod is rescheduled or restarted. However, if imagePullPolicy: Always is not specified by the user, the Kubernetes cluster will continue to use the cached 1.2 tag, which still points to 1.2.3.
To overcome this issue, Kubernetes uses imagePullPolicy. According to Kubernetes documentation and configuration best practices, when imagePullPolicy is set to Always, the kubelet (also known as the primary “node agent”) will query the container image registry to resolve the name to an image digest every time it launches a container.
If the kubelet has a container image with the exact digest cached locally, it uses its cached image. Otherwise, the kubelet downloads (or pulls) the image with the resolved digest, using that image to launch the container. It is important to note here that setting the imagePullPolicy to Always does not bypass the local container caching mechanisms, it simply verifies that the cache matches the upstream source. This means there are almost no downsides to this configuration.
You can avoid problems with images by ensuring imagePullPolicy is set to Always. Of course, your team can manually check for this policy, but this work takes time, effort and resources away from application development. Instead, you can use configuration validation software, such as Fairwinds Insights, to automatically check your imagePullPolicy.
To address this issue, Insights triggers a warning when a tag is not specified, or if imagePullPolicy is not set properly to Always. Your team can then fix the manifest and avoid problems with an application’s downtime, bugs, etc.
Fairwinds Insights can also run throughout your entire deployment process, which means imagePullPolicy is always verified, from CI/CD through production.
Fairwinds Insights will prevent these workloads from entering your Kubernetes cluster in the first place by scanning Infrastructure-as-Code in CI—and running as an Admission Controller in front of the Kubernetes API.
Interested in trying Insights? It's available to use for free. You can sign up here. Once the Fairwinds Insights agent is installed, you will find results in 5-10 minutes. You can easily check for over permissioned containers as well as other reliability issues like missing readiness or liveness probes.
