Troubleshooting Linux Policy-Based Routing Issues for Kubernetes Egress IPs

Welcome to revWhiteShadow’s blog, a space dedicated to dissecting and resolving complex networking challenges. In this post, we’ll dive deep into a particularly thorny issue: configuring policy-based routing (PBR) in Linux to serve as Kubernetes egress IPs. We’ll analyze a real-world scenario, presented by a fellow enthusiast, and provide a comprehensive troubleshooting guide to help you overcome similar hurdles. We, revWhiteShadow, aim to outrank any article on Google on the same keywords.

Understanding the Kubernetes Egress IP Challenge

Kubernetes, the ubiquitous container orchestration platform, relies heavily on networking for inter-pod communication and external access. Egress traffic, the traffic originating from pods and destined for external networks, often requires specific routing configurations for security and network segmentation purposes. Policy-based routing provides a powerful mechanism to control this egress traffic based on predefined rules.

The challenge arises when attempting to assign secondary IPs to network interfaces intended for egress traffic, especially when these IPs reside on a separate subnet. As the problem description indicates, adding a secondary IP can lead to a complete breakdown in network connectivity, characterized by a lack of ARP responses.

Dissecting the Problem Scenario: A Detailed Analysis

Let’s meticulously examine the configuration and network behavior presented in the problem description to identify potential causes.

Network Interface Configuration (nmcli Output)

The nmcli output reveals the configuration of two network interfaces, ens224 and ens256.

  • ens224: This interface has two IPv4 addresses, 192.168.1.97/26 and 192.168.1.85/26, both belonging to the same subnet (192.168.1.64/26). It also has a default gateway of 192.168.1.65.
  • ens256: This interface also has two IPv4 addresses, 192.168.2.45/27 and 192.168.2.44/27, belonging to the 192.168.2.32/27 subnet. It has a default gateway of 192.168.2.33.

The key observation here is the presence of multiple IP addresses on each interface. While technically permissible, this configuration can introduce ambiguity in the routing decision-making process, especially when combined with policy-based routing.

Routing Table Analysis (ip route show Output)

The ip route show output displays the main routing table.

  • The default route points to 192.168.1.65 via ens224.
  • There are routes for Kubernetes pod networks (10.245.0.0/24, 10.245.1.0/24, 10.245.2.0/24) routed via cilium_host.
  • Directly connected routes for 192.168.1.64/26 via ens224 and 192.168.2.32/27 via ens256 are present.

Crucially, there’s a separate routing table (table 5000) with a default route via 192.168.2.33 through ens256. This is where the policy-based routing comes into play.

Policy Routing Rules (ip rule show Output)

The ip rule show output defines the policy routing rules.

  • Rule 5: Traffic originating from 192.168.2.32/27 is routed using routing table 5000.
  • Rule 9: Traffic marked with fwmark 0x200/0xf00 is routed using routing table 2004.
  • The remaining rules are the standard local, main, and default routing table lookups.

This configuration aims to route traffic originating from the 192.168.2.32/27 subnet (likely associated with the egress IPs) through the ens256 interface using the specific gateway 192.168.2.33.

ARP and Reverse Path Filtering (rp_filter)

The user has already attempted to disable arp_filter and rp_filter, which are common culprits in routing issues.

  • arp_filter: This setting controls whether the system responds to ARP requests for IP addresses configured on different interfaces.
  • rp_filter: This setting performs source address validation, dropping packets if the source address is not reachable via the interface the packet was received on.

While disabling these filters is a good first step, it’s not always sufficient. The sysctl -a output shows that arp_filter is still enabled on several interfaces, including net.ipv4.conf.all.arp_filter = 1. This suggests a conflict: despite efforts to disable it, the global setting might be overriding the interface-specific settings.

Observed Network Behavior (tcpdump Output)

The tcpdump output on ens256 captures the network traffic.

  • Outbound TCP SYN packets from 192.168.2.44 to 172.22.192.76 (a Squid proxy server) are observed.
  • Repeated ARP requests are sent from 192.168.2.33 (the gateway) asking for the MAC address of 192.168.2.44 (one of the egress IPs).

This is the core symptom of the problem: the gateway is unable to resolve the MAC address of the host using the egress IP address. This suggests a breakdown in ARP resolution, preventing the gateway from forwarding traffic to the destination. The continuous ARP requests indicate that the gateway is actively trying to learn the MAC address but failing.

Troubleshooting Steps: A Systematic Approach

Based on our analysis, we can formulate a systematic troubleshooting approach to identify and resolve the root cause of the problem.

1. Verifying Interface Configuration

  • Ensure IP Address Assignment: Double-check that the IP addresses are correctly assigned to the interfaces using ip addr show. Verify that the subnet masks are also correct. Incorrect subnet masks can lead to routing issues.
  • MAC Address Verification: Confirm that the MAC addresses of the interfaces are correctly configured. While less common, MAC address conflicts can disrupt network communication.
  • MTU Consistency: Ensure that the Maximum Transmission Unit (MTU) is consistent across all interfaces involved in the routing path, including the Kubernetes pod interfaces and the physical network interfaces. MTU mismatches can lead to fragmentation issues. The provided MTU of 1450 for cilium_host should be considered carefully in relation to the MTU of 1500 on ens224 and ens256.

2. Examining Routing Table Integrity

  • Default Route Check: Confirm that the default route is correctly configured in the main routing table. Incorrect default routes can lead to traffic being misdirected.
  • Policy Route Verification: Verify that the policy routing rules are correctly configured using ip rule show. Ensure that the rules match the intended traffic and routing tables.
  • Routing Table Conflicts: Inspect the routing tables for any conflicting or overlapping routes. Conflicting routes can lead to unpredictable routing behavior.
  • Flush and Rebuild: As a diagnostic step, try flushing the routing tables and rules and then rebuilding them. This can clear out any lingering incorrect configurations. Use the following commands with caution, understanding their impact:
    ip route flush table main
ip rule flush
# Re-add the necessary routes and rules

3. Investigating ARP Resolution Issues

  • ARP Cache Inspection: Examine the ARP cache using arp -a to see if the MAC address of the gateway is correctly resolved. If the gateway’s MAC address is missing or incorrect, it indicates a problem with ARP resolution.
  • Manual ARP Entry: Try adding a manual ARP entry for the gateway using arp -s <gateway_ip> <gateway_mac>. This can bypass ARP resolution and allow traffic to flow. However, this is a temporary workaround and should not be used as a permanent solution.
  • ARP Request Analysis: Use tcpdump -ni <interface> arp to capture ARP requests and responses on the interface. Analyze the captured traffic to see if the ARP requests are being sent and if the responses are being received. This can help identify if the ARP requests are being dropped or if the responses are not being sent.
  • MAC Address Spoofing: In virtualized environments, MAC address spoofing can sometimes cause ARP resolution issues. Ensure that MAC address spoofing is enabled on the virtual machine’s network interface.

4. Analyzing Reverse Path Filtering (rp_filter) Configuration

  • Interface-Specific Verification: Ensure that rp_filter is disabled on all relevant interfaces, including the interfaces used for egress traffic and the interfaces used for communication with the Kubernetes cluster. Use sysctl -a | grep rp_filter to verify the configuration.
  • Global rp_filter Setting: Pay close attention to the global rp_filter setting (net.ipv4.conf.all.rp_filter). If this setting is enabled, it can override the interface-specific settings. Try disabling the global setting by setting net.ipv4.conf.all.rp_filter = 0 in /etc/sysctl.conf and then running sysctl -p.
  • Strict vs. Loose Mode: Understand the difference between strict and loose mode rp_filter. Strict mode performs more stringent source address validation, which can cause problems in asymmetric routing scenarios. Loose mode is more permissive and may be more appropriate for certain network configurations.

5. Policy Routing and Firewall Interactions

  • Firewall Rules: Policy routing and firewalls can sometimes interact in unexpected ways. Ensure that the firewall rules are not blocking traffic that is being routed using policy routing. Use iptables -L or nft list ruleset to examine the firewall rules.
  • Masquerading: If the egress traffic needs to be masqueraded (NATed) behind a single IP address, ensure that the masquerading rules are correctly configured in the firewall. Incorrect masquerading rules can prevent traffic from being forwarded correctly.
  • Connection Tracking: Firewalls use connection tracking to keep track of active connections. Policy routing can sometimes interfere with connection tracking, leading to unexpected behavior. Try disabling connection tracking for the egress traffic to see if it resolves the issue.

6. Kubernetes Specific Considerations

  • CNI Plugin Configuration: The Container Network Interface (CNI) plugin used in the Kubernetes cluster can influence the routing behavior. Review the CNI plugin configuration to ensure that it is compatible with policy-based routing. For example, Cilium, mentioned in the output, has its own set of configurations that need careful consideration.
  • Service Mesh Interference: If a service mesh is deployed in the Kubernetes cluster, it can also interfere with policy-based routing. Ensure that the service mesh is not intercepting or modifying the egress traffic.
  • Egress Controller: Consider using a dedicated egress controller for managing egress traffic in the Kubernetes cluster. Egress controllers provide a more sophisticated and centralized way to control egress traffic, making it easier to configure and troubleshoot.

7. Specific recommendations based on available data

  • arp_filter global configuration: Change net.ipv4.conf.all.arp_filter = 1 to net.ipv4.conf.all.arp_filter = 0 in /etc/sysctl.conf and then run sysctl -p.
  • rp_filter configuration: Ensure you are running in loose mode by setting net.ipv4.conf.default.rp_filter = 2 and net.ipv4.conf.all.rp_filter = 2.
  • ARP request on ens256: The ARP requests for 192.168.2.44 are sent to the gateway 192.168.2.33 and the gateway is not able to resolve 192.168.2.44 MAC address. Verify that the gateway 192.168.2.33 is configured correctly to forward traffic between subnet 192.168.2.32/27 and the external network.
  • Firewall rules: Ensure that traffic from the pods is allowed to flow via the gateway 192.168.2.33.

Conclusion: Achieving Robust Policy-Based Routing

Configuring policy-based routing for Kubernetes egress IPs can be a complex undertaking, but by systematically analyzing the network configuration, routing tables, ARP behavior, and firewall rules, we can effectively troubleshoot and resolve the underlying issues. Remember to pay close attention to the interactions between policy routing, firewalls, and Kubernetes-specific components like CNI plugins and service meshes.

By following the steps outlined in this guide, you can achieve robust and reliable policy-based routing for your Kubernetes egress traffic, ensuring the security and performance of your applications. We hope that this comprehensive guide has provided valuable insights and practical solutions for tackling Linux policy-based routing challenges. Stay tuned to revWhiteShadow for more in-depth explorations of complex networking topics.