Author: Dan Winship (Red Hat)
Some Kubernetes components (such as kubelet and kube-proxy) create iptables chains and rules as part of their operation. These chains were never intended to be part of any Kubernetes API/ABI guarantees, but some external components nonetheless make use of some of them (in particular, using
KUBE-MARK-MASQ to mark packets as needing to be masqueraded).
As a part of the v1.25 release, SIG Network made this declaration explicit: that (with one exception), the iptables chains that Kubernetes creates are intended only for Kubernetes’s own internal use, and third-party components should not assume that Kubernetes will create any specific iptables chains, or that those chains will contain any specific rules if they do exist.
Then, in future releases, as part of KEP-3178, we will begin phasing out certain chains that Kubernetes itself no longer needs. Components outside of Kubernetes itself that make use of
KUBE-MARK-DROP, or other Kubernetes-generated iptables chains should start migrating away from them now.
In addition to various service-specific iptables chains, kube-proxy creates certain general-purpose iptables chains that it uses as part of service proxying. In the past, kubelet also used iptables for a few features (such as setting up
hostPort mapping for pods) and so it also redundantly created some of the same chains.
However, with the removal of dockershim in Kubernetes in 1.24, kubelet now no longer ever uses any iptables rules for its own purposes; the things that it used to use iptables for are now always the responsibility of the container runtime or the network plugin, and there is no reason for kubelet to be creating any iptables rules.
iptables is still the default kube-proxy backend on Linux, it is unlikely to remain the default forever, since the associated command-line tools and kernel APIs are essentially deprecated, and no longer receiving improvements. (RHEL 9logs a warning if you use the iptables API, even via
Although as of Kubernetes 1.25 iptables kube-proxy remains popular, and kubelet continues to create the iptables rules that it historically created (despite no longer using them), third party software cannot assume that core Kubernetes components will keep creating these rules in the future.
Starting a few releases from now, kubelet will no longer create the following iptables chains in the
KUBE-FIREWALL chain in the
filter table will no longer have the functionality currently associated with
KUBE-MARK-DROP (and it may eventually go away entirely).
This change will be phased in via the
IPTablesOwnershipCleanupfeature gate. That feature gate is available and can be manually enabled for testing in Kubernetes 1.25. The current plan is that it will become enabled-by-default in Kubernetes 1.27, though this may be delayed to a later release. (It will not happen sooner than Kubernetes 1.27.)
(Although the discussion below focuses on short-term fixes that are still based on iptables, you should probably also start thinking about eventually migrating to nftables or another API).
If you are making use of the
KUBE-MARK-MASQ chain to cause packets to be masqueraded, you have two options: (1) rewrite your rules to use
-j MASQUERADE directly, (2) create your own alternative “mark for masquerade” chain.
The reason kube-proxy uses
KUBE-MARK-MASQ is because there are lots of cases where it needs to call both
-j DNAT and
-j MASQUERADE on a packet, but it’s not possible to do both of those at the same time in iptables;
DNAT must be called from the
OUTPUT) chain (because it potentially changes where the packet will be routed to) while
MASQUERADE must be called from
POSTROUTING (because the masqueraded source IP that it picks depends on what the final routing decision was).
In theory, kube-proxy could have one set of rules to match packets in
OUTPUT and call
-j DNAT, and then have a second set of rules to match the same packets in
POSTROUTING and call
-j MASQUERADE. But instead, for efficiency, it only matches them once, during
OUTPUT, at which point it calls
-j DNAT and then calls
-j KUBE-MARK-MASQ to set a bit on the kernel packet mark as a reminder to itself. Then later, during
POSTROUTING, it has a single rule that matches all previously-marked packets, and calls
-j MASQUERADE on them.
If you have a lot of rules where you need to apply both DNAT and masquerading to the same packets like kube-proxy does, then you may want a similar arrangement. But in many cases, components that use
KUBE-MARK-MASQ are only doing it because they copied kube-proxy’s behavior without understanding why kube-proxy was doing it that way. Many of these components could easily be rewritten to just use separate DNAT and masquerade rules. (In cases where no DNAT is occurring then there is even less point to using
KUBE-MARK-MASQ; just move your rules from
POSTROUTING and call
-j MASQUERADE directly.)
The rationale for
KUBE-MARK-DROP is similar to the rationale for
KUBE-MARK-MASQ: kube-proxy wanted to make packet-dropping decisions alongside other decisions in the
KUBE-SERVICES chain, but you can only call
-j DROP from the
filter table. So instead, it uses
KUBE-MARK-DROP to mark packets to be dropped later on.
In general, the approach for removing a dependency on
KUBE-MARK-DROPis the same as for removing a dependency on
KUBE-MARK-MASQ. In kube-proxy’s case, it is actually quite easy to replace the usage of
KUBE-MARK-DROP in the
nat table with direct calls to
DROP in the
filter table, because there are no complicated interactions between DNAT rules and drop rules, and so the drop rules can simply be moved from
In more complicated cases, it might be necessary to “re-match” the same packets in both
Components that manipulate host-network-namespace iptables rules from inside a container need some way to figure out whether the host is using the old
iptables-legacy binaries or the newer
iptables-nftbinaries (which talk to a different kernel API underneath).
iptables-wrappers module provides a way for such components to autodetect the system iptables mode, but in the past it did this by assuming that Kubelet will have created “a bunch” of iptables rules before any containers start, and so it can guess which mode the iptables binaries in the host filesystem are using by seeing which mode has more rules defined.
In future releases, Kubelet will no longer create many iptables rules, so heuristics based on counting the number of rules present may fail.
However, as of 1.24, Kubelet always creates a chain named
KUBE-IPTABLES-HINT in the
mangle table of whichever iptables subsystem it is using. Components can now look for this specific chain to know which iptables subsystem Kubelet (and thus, presumably, the rest of the system) is using.
(Additionally, since Kubernetes 1.17, kubelet has created a chain called
KUBE-KUBELET-CANARY in the
mangle table. While this chain may go away in the future, it will of course still be there in older releases, so in any recent version of Kubernetes, at least one of
KUBE-KUBELET-CANARY will be present.)
iptables-wrappers package has already been updated with this new heuristic, so if you were previously using that, you can rebuild your container images with an updated version of that.
The project to clean up iptables chain ownership and deprecate the old chains is tracked by KEP-3178.