Mastering Network Routing: Fixing eth0 and eth1 Traffic Failures with Advanced IP Rules

At revWhiteShadow, we understand the critical importance of a robust and precisely configured network. When traffic fails to route correctly through specific interfaces like eth0 while eth1 remains operational, it signifies a misconfiguration within your routing policies. This guide is meticulously crafted to address precisely this scenario: troubleshooting and resolving outbound traffic failures on eth0 when eth1 is functioning as expected, particularly after implementing advanced ip rule add and ip route add commands to manage multiple public IP addresses and custom routing tables.

We acknowledge the specific configuration you’ve implemented, involving two public IPs, each routed through separate routers acting as DMZs. Your goal is to ensure that traffic originating from or destined for a specific IP address is correctly directed via eth0, without disrupting the already functional routing via eth1. The provided commands illustrate an attempt to create policy-based routing, where traffic associated with 10.0.0.108 is directed to table 1, and traffic associated with 10.0.0.100 is directed to table 2.

# Initial configuration attempts for routing based on source IP
ip rule add from 10.0.0.108/32 table 1 # outbound
ip rule add to 10.0.0.108/32 table 1   # inbound
ip route add default via 10.0.0.1 dev eth0 table 1

ip rule add from 10.0.0.100/32 table 2 # outbound
ip rule add to 10.0.0.100/32 table 2   # inbound
ip route add default via 10.0.0.1 dev eth1 table 2

The observed behavior, where ping to 8.8.8.8 from eth0 fails with “Destination Host Unreachable” while eth1 successfully pings, indicates a fundamental issue in how your system is evaluating and selecting routes for traffic originating from the eth0 interface or associated with its IP. This could stem from several factors, including incorrect rule priorities, missing default routes in the intended table, or conflicts in the IP routing policy database.

Our objective is to meticulously dissect these commands and the underlying principles of Linux advanced routing to achieve your desired network behavior. We will focus on ensuring that traffic originating from 10.0.0.108 correctly egresses via eth0, while preserving the integrity of the eth1 interface’s functionality.

Understanding Linux Advanced Routing and IP Rules

Before delving into specific troubleshooting steps, it’s crucial to grasp the mechanics of Linux advanced routing. Unlike traditional routing, which relies solely on destination IP addresses, policy-based routing (or source-based routing) allows for routing decisions to be made based on various criteria, such as the source IP address, ingress interface, ToS (Type of Service) bits, and more. This is achieved through the use of IP rules and routing tables.

The IP routing policy database (RPDB) is consulted sequentially. Each entry in the RPDB is an IP rule. When the kernel needs to route a packet, it iterates through the rules in order of priority. The first rule that matches the packet’s criteria determines which routing table is consulted.

Key components involved:

ip rule: This command is used to manage the RPDB. You use it to add, delete, or list rules that define matching criteria and the corresponding routing table to use.
- from <address/mask>: Matches packets originating from a specific source IP address.
- to <address/mask>: Matches packets destined for a specific destination IP address.
- table <table_id>: Specifies which routing table to consult if the rule matches.
- priority <number>: Assigns a priority to the rule. Lower numbers indicate higher priority.
ip route: This command is used to manage the routing tables themselves. You can add, delete, or list routes within specific tables.
- default via <gateway_ip> dev <interface>: Defines the default gateway for a particular routing table.
- table <table_id>: Specifies the routing table to which the route belongs.

When a packet arrives, the kernel checks the RPDB:

It starts with the lowest priority rule.
If the rule’s criteria match the packet (e.g., source IP, destination IP), the kernel consults the routing table specified in that rule.
If no rule matches, or if the consulted table doesn’t have a route for the destination, the kernel falls back to the default routing table (usually main).

The problem you are encountering likely stems from how your rules interact with each other and the default routing mechanisms.

Diagnosing the `eth0` Traffic Failure: A Step-by-Step Analysis

The “Destination Host Unreachable” error typically indicates that a router (in this case, possibly your own system’s routing logic or the next hop) cannot find a path to the destination. Given that eth1 is working, the issue is specifically with the routing path for eth0.

Let’s break down the provided configuration and potential issues:

1. Understanding Your Current Rules and Tables

Table 1 (for 10.0.0.108):
- ip rule add from 10.0.0.108/32 table 1
- ip rule add to 10.0.0.108/32 table 1
- ip route add default via 10.0.0.1 dev eth0 table 1
This setup attempts to ensure that traffic originating from 10.0.0.108 and traffic destined for 10.0.0.108 should use table 1. The default route in table 1 points to 10.0.0.1 via eth0.
Table 2 (for 10.0.0.100):
- ip rule add from 10.0.0.100/32 table 2
- ip rule add to 10.0.0.100/32 table 2
- ip route add default via 10.0.0.1 dev eth1 table 2
This setup aims to route traffic originating from 10.0.0.100 and traffic destined for 10.0.0.100 via table 2, with a default route pointing to 10.0.0.1 via eth1.

2. Potential Conflicts and Missing Configurations

Rule Priority: When you add rules without specifying a priority, they are added with a default priority. If the new rules have higher priority (lower number) than existing rules that might be directing traffic elsewhere, they can cause conflicts. It’s crucial to manage priorities to ensure your custom rules are evaluated correctly.
Local Routes: For traffic originating from 10.0.0.108 on eth0, the system needs to know that 10.0.0.108 itself is reachable via eth0. Without a local route pointing to the interface, outbound traffic might not be correctly associated with the source IP and interface.
Inbound Routing: The ip rule add to ... rules are for inbound traffic. For inbound traffic to work correctly, the system must know that the destination IP (10.0.0.108 or 10.0.0.100) is present on the respective interface. This is usually handled by the interface’s primary IP configuration.
Default Gateway Reachability: The ping failure suggests that the gateway 10.0.0.1 (which is likely the IP address of your router on the eth0 side) is not reachable or not correctly configured as the next hop for the default route in table 1.
Implicit local and main tables: The system always has implicit rules that use the local and main routing tables. Your custom rules are inserted into the RPDB, and their priority determines when they are evaluated relative to these implicit rules.

3. Verifying the Current Routing State

Before making changes, it is imperative to understand the current state of your routing policies and tables.

3.1. Examining IP Rules (`ip rule show`)

This command displays the entire IP routing policy database. You should pay close attention to the priority of each rule.

ip rule show

Expected Output Snippet (example):

0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

Your custom rules will appear interspersed within this list, typically with priorities between 0 and 32766.

3.2. Examining Routing Tables (`ip route show table <table_id>`)

Check the contents of your custom routing tables.

For Table 1:

ip route show table 1

Expected Output Snippet (example, if correctly configured):

default via 10.0.0.1 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.108

For Table 2:

ip route show table 2

Expected Output Snippet (example, if correctly configured):

default via 10.0.0.1 dev eth1
10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.100

For the main table (default):

ip route show table main

This will show your system’s primary routing table.

3.3. Examining Interface Information (`ip addr show`)

Ensure that eth0 and eth1 are correctly configured with their respective IP addresses and that they are up.

ip addr show

Expected Output Snippet (example):

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.108/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.100/24 brd 10.0.0.255 scope global eth1
       valid_lft forever preferred_lft forever

(Note: The /24 subnet mask might be illustrative; adjust according to your actual network configuration.)

Strategies to Fix `eth0` Outbound Traffic Failures

We will now focus on providing the precise steps to resolve the eth0 issue while safeguarding the functionality of eth1. The core of the solution lies in ensuring that the IP rules are correctly prioritized and that the routing tables contain all necessary entries for proper packet forwarding.

### 1. Prioritizing Rules for Specific Source IPs

The ip rule add command, by default, adds rules at a priority that might not be correct relative to other existing rules. It is best practice to explicitly define priorities to avoid conflicts. We want your custom rules to be evaluated before the general main table rules for traffic associated with your specific IPs.

Let’s assume the default main table rules have a priority of 32766. We should assign your custom rules a priority lower than this, for example, 100 for eth0 and 101 for eth1 to ensure they take precedence.

Remove existing rules (if you want to start clean and re-apply with correct priorities):

# For table 1
ip rule del from 10.0.0.108/32 table 1
ip rule del to 10.0.0.108/32 table 1

# For table 2
ip rule del from 10.0.0.100/32 table 2
ip rule del to 10.0.0.100/32 table 2

Add rules with specific priorities:

1.1. Rules for `eth0` (Table 1 - Priority 100)

These rules will direct traffic matching 10.0.0.108 to table 1.

# Outbound traffic from 10.0.0.108 uses table 1
ip rule add from 10.0.0.108/32 table 1 priority 100

# Inbound traffic to 10.0.0.108 uses table 1
ip rule add to 10.0.0.108/32 table 1 priority 100

1.2. Rules for `eth1` (Table 2 - Priority 101)

These rules direct traffic matching 10.0.0.100 to table 2. We assign a slightly lower priority (higher number) than eth0’s rules, ensuring that if there were any overlap in source/destination IPs, eth0 would take precedence.

# Outbound traffic from 10.0.0.100 uses table 2
ip rule add from 10.0.0.100/32 table 2 priority 101

# Inbound traffic to 10.0.0.100 uses table 2
ip rule add to 10.0.0.100/32 table 2 priority 101

### 2. Ensuring Correct Routing Table Entries

The failure in eth0 might be due to the absence of a proper default route or local route within table 1.

2.1. Verifying and Correcting Table 1 Default Route

The command ip route add default via 10.0.0.1 dev eth0 table 1 is crucial. Let’s ensure it’s correctly in place and that 10.0.0.1 is indeed the correct gateway IP for eth0.

Add/Confirm the default route for table 1:

# Assuming 10.0.0.1 is the gateway IP on the eth0 segment
ip route add default via 10.0.0.1 dev eth0 table 1

2.2. Adding a Local Route for the Source IP in Table 1

This is a critical step that is often overlooked. For traffic originating from 10.0.0.108, the system needs to know that this IP address is directly reachable via eth0. Without this, the ip rule add from 10.0.0.108 might not correctly associate the outgoing packet with eth0.

Add a local route for 10.0.0.108 in table 1:

# Assuming your eth0 interface has the IP 10.0.0.108 with a /24 subnet mask
ip route add 10.0.0.108/24 dev eth0 src 10.0.0.108 table 1

This command explicitly states that the network 10.0.0.108/24 is reachable via eth0 and that packets sourced from 10.0.0.108 should use this interface.

2.3. Verifying and Correcting Table 2 Default Route (for `eth1`)

While eth1 is working, it’s good practice to ensure its configuration is also robust.

Add/Confirm the default route for table 2:

# Assuming 10.0.0.1 is the gateway IP on the eth1 segment (this might be different from eth0's gateway if they are separate routers)
# If the gateway IP for eth1 is DIFFERENT from eth0's gateway IP, replace 10.0.0.1 with the correct gateway for eth1.
# For example, if the eth1 gateway is 10.0.1.1:
# ip route add default via 10.0.1.1 dev eth1 table 2
ip route add default via 10.0.0.1 dev eth1 table 2

2.4. Adding a Local Route for the Source IP in Table 2

Similarly, for eth1 and its associated IP.

Add a local route for 10.0.0.100 in table 2:

# Assuming your eth1 interface has the IP 10.0.0.100 with a /24 subnet mask
ip route add 10.0.0.100/24 dev eth1 src 10.0.0.100 table 2

### 3. Addressing Potential ARP and Gateway Issues

The “Destination Host Unreachable” can also occur if the system cannot resolve the MAC address of the gateway (10.0.0.1 in your case for eth0) via ARP, or if the gateway itself is not responding.

3.1. Forcing an ARP Request

While ping usually triggers an ARP request, sometimes it’s useful to explicitly populate the ARP cache.

arping -I eth0 10.0.0.1

This command sends ARP requests directly out of eth0 to 10.0.0.1. If you receive replies, your ARP is working.

3.2. Checking Gateway Status

Ensure that your router on the eth0 side is powered on, connected, and correctly routing traffic. The IP address 10.0.0.1 should be reachable from your server’s 10.0.0.108 IP.

### 4. Comprehensive Configuration Summary and Verification

After applying the above steps, we should have a robust policy-based routing setup.

4.1. Final Check of IP Rules

ip rule show

You should see rules similar to this (priorities might vary slightly based on existing system rules, but the critical ones should be prioritized):

0:	from all lookup local
100:	from 10.0.0.108 lookup 1
100:	to 10.0.0.108 lookup 1
101:	from 10.0.0.100 lookup 2
101:	to 10.0.0.100 lookup 2
... (other system rules) ...
32766:	from all lookup main
32767:	from all lookup default

4.2. Final Check of Routing Tables

Table 1:

ip route show table 1

Expected output:

default via 10.0.0.1 dev eth0
10.0.0.108/24 dev eth0 scope link src 10.0.0.108

Table 2:

ip route show table 2

Expected output:

default via 10.0.0.1 dev eth1 # Or the correct gateway for eth1 if different
10.0.0.100/24 dev eth1 scope link src 10.0.0.100

4.3. Testing the Configuration

Now, re-test your ping from both interfaces.

Test eth0:

ping -I eth0 8.8.8.8

You should see successful pings originating from 10.0.0.108 via eth0.

Test eth1:

ping -I eth1 8.8.8.8

This should continue to work as before.

4.4. Handling Other Traffic Types

Remember that the ip rule add to <IP> commands are for inbound traffic. For these to be effective, your system must be configured to accept connections on 10.0.0.108 via eth0. This typically means the IP address is correctly assigned to the interface.

If you have other servers or services running on your machine that need to use specific outbound interfaces based on their source IP, you would extend this pattern. For instance, if a process on your server uses the IP 10.0.0.108 to initiate a connection, the ip rule add from 10.0.0.108 would direct that traffic to table 1, and the ip route add default via 10.0.0.1 dev eth0 table 1 would ensure it leaves via eth0.

If you are running multiple applications and want to control which application uses which interface, you might need to go further and use iptables to mark packets and then use ip rule with the fwmark option. However, based on your description, controlling traffic based on source IP should suffice.

### 5. Persistence of Configuration

These ip commands are ephemeral and will be lost upon reboot. To make them permanent, you need to configure your network interface scripts or use a systemd service.

Debian/Ubuntu (using /etc/network/interfaces): You can add post-up commands to your interface stanzas.
CentOS/RHEL (using ifcfg-* files or NetworkManager): NetworkManager dispatchers or custom scripts executed on interface up events can be used.
Systemd networkd: Create .network files with the appropriate [RoutingPolicy] and [Route] sections.
Custom Systemd Service: Create a service that runs these ip commands at boot.

Example for a simple systemd service:

Create /etc/systemd/system/policy-routing.service:

[Unit]
Description=Apply custom IP routing policies
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/apply-policy-routing.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Create /usr/local/bin/apply-policy-routing.sh and make it executable (chmod +x):

#!/bin/bash

# Add rules for eth0
ip rule add from 10.0.0.108/32 table 1 priority 100
ip rule add to 10.0.0.108/32 table 1 priority 100

# Add routes for table 1
ip route add default via 10.0.0.1 dev eth0 table 1
ip route add 10.0.0.108/24 dev eth0 src 10.0.0.108 table 1

# Add rules for eth1
ip rule add from 10.0.0.100/32 table 2 priority 101
ip rule add to 10.0.0.100/32 table 2 priority 101

# Add routes for table 2
ip route add default via 10.0.0.1 dev eth1 table 2 # Adjust gateway if needed for eth1
ip route add 10.0.0.100/24 dev eth1 src 10.0.0.100 table 2

exit 0

Then enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable policy-routing.service
sudo systemctl start policy-routing.service

Conclusion

By meticulously defining IP rules with appropriate priorities and ensuring that each routing table contains the necessary default and local routes, we have effectively addressed the outbound traffic failure on eth0. The addition of local routes for the source IP addresses within their respective policy tables is often the missing piece in such configurations, enabling the kernel to correctly associate packets with the intended egress interface.

At revWhiteShadow, our commitment is to provide detailed, actionable guidance that empowers you to manage your network infrastructure with precision. This solution ensures that your dual-homed setup operates flawlessly, with traffic correctly segregated and routed according to your specific requirements, all while preserving the existing functionality of your working interfaces.

How to fix this eth0 and eth1 where traffic going out via eth0 is failing?

Mastering Network Routing: Fixing eth0 and eth1 Traffic Failures with Advanced IP Rules #

Understanding Linux Advanced Routing and IP Rules #

Diagnosing the eth0 Traffic Failure: A Step-by-Step Analysis #

1. Understanding Your Current Rules and Tables #

2. Potential Conflicts and Missing Configurations #

3. Verifying the Current Routing State #

3.1. Examining IP Rules (ip rule show) #

3.2. Examining Routing Tables (ip route show table <table_id>) #

3.3. Examining Interface Information (ip addr show) #

Strategies to Fix eth0 Outbound Traffic Failures #

### 1. Prioritizing Rules for Specific Source IPs #

1.1. Rules for eth0 (Table 1 - Priority 100) #

1.2. Rules for eth1 (Table 2 - Priority 101) #

### 2. Ensuring Correct Routing Table Entries #

2.1. Verifying and Correcting Table 1 Default Route #

2.2. Adding a Local Route for the Source IP in Table 1 #

2.3. Verifying and Correcting Table 2 Default Route (for eth1) #

2.4. Adding a Local Route for the Source IP in Table 2 #

### 3. Addressing Potential ARP and Gateway Issues #

3.1. Forcing an ARP Request #

3.2. Checking Gateway Status #

### 4. Comprehensive Configuration Summary and Verification #

4.1. Final Check of IP Rules #

4.2. Final Check of Routing Tables #

4.3. Testing the Configuration #

4.4. Handling Other Traffic Types #

### 5. Persistence of Configuration #

Conclusion #