Mastering KVM/QEMU Libvirt Network “default” NAT: Solving Guest Internet Connectivity Issues

At revWhiteShadow, we understand the critical importance of seamless network connectivity for your virtualized environments. When your KVM/QEMU guests, configured with libvirt’s default NAT network, can communicate perfectly with the host and vice versa, but inexplicably fail to reach the external internet, it presents a significant hurdle. This article provides a comprehensive deep dive into diagnosing and rectifying such issues, ensuring your virtual machines can freely access the global network. We will meticulously examine the configuration, routing, and firewall rules that govern libvirt’s default NAT, offering precise solutions to overcome these connectivity barriers.

Understanding the KVM/QEMU Libvirt “default” NAT Network Architecture

Libvirt’s default network, often named “default,” is a powerful and convenient abstraction for managing virtual machine networking. It typically employs Network Address Translation (NAT) to allow multiple virtual machines to share a single IP address on the host’s physical network. This setup involves several key components:

The Bridge Interface (virbr0): Libvirt creates a virtual bridge interface, commonly named virbr0, on the host system. This bridge acts as the central point of connection for all virtual machines attached to the default network. The virbr0 interface is assigned an IP address within a private subnet, typically 192.168.122.1/24.
NAT (Network Address Translation): When a guest VM initiates a connection to the external internet, the NAT process rewrites the source IP address of the outgoing packets from the guest’s private IP (e.g., 192.168.122.177) to the IP address of the host’s primary network interface (e.g., enp3s0). This makes the traffic appear to originate from the host itself.
DHCP Server: Libvirt also runs a DHCP server on the virbr0 interface, responsible for assigning IP addresses, subnet masks, default gateways, and DNS server information to the connected guest VMs. In the “default” network, this typically assigns IPs from the 192.168.122.2 to 192.168.122.254 range, with 192.168.122.1 as the default gateway.
iptables Rules: The core of the NAT functionality, along with necessary filtering and forwarding rules, is managed by iptables (or nftables on newer systems). These rules dictate how traffic flows between the guest VMs, the host, and the external network.

When a guest VM can reach the host and the virbr0 interface, but not the external internet, it strongly suggests that the routing or firewalling rules are not correctly facilitating the NAT and forwarding of traffic to the host’s upstream network.

Diagnosing the Root Cause: A Step-by-Step Approach

To effectively resolve the internet connectivity issue for your KVM/QEMU guest using the default NAT network, we must systematically investigate potential points of failure.

#### 1. Verifying Network Configuration (`virsh net-dumpxml default`)**

Your provided output for $ sudo virsh net-dumpxml default shows a standard and generally correct configuration:

<network connections='1'>
  <name>default</name>
  <uuid>f1eff8aa-73e7-4573-8d36-571a85714777</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:73:62:06'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

This confirms:

The network is named “default.”
It is configured for NAT forwarding.
A bridge named virbr0 is used.
The virbr0 bridge has the IP 192.168.122.1 with a 255.255.255.0 subnet mask.
A DHCP range is defined for guest IPs.

This part of the configuration appears sound.

#### 2. Examining Host Network Interface (`ip address show dev virbr0`)**

The output $ ip address show dev virbr0 also confirms the expected setup:

7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:73:62:06 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

This shows that virbr0 is up, has the correct IP address, and is properly configured.

#### 3. Guest IP Configuration (`ipconfig` in Windows)**

The guest’s IP configuration:

Ethernet adapter Local Area Connection 2:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::68b4:6322:b7d9:e1b%13
   IPv4 Address. . . . . . . . . . . : 192.168.122.177
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.122.1

This indicates that the guest has received a valid IP address from the libvirt DHCP server, has the correct subnet mask, and most importantly, the default gateway is correctly set to 192.168.122.1, which is the virbr0 interface. This confirms that the guest knows where to send traffic destined for external networks.

#### 4. Analyzing `iptables` Rules (`sudo iptables-save`)**

The iptables-save output is crucial for understanding how traffic is being handled. We need to pay close attention to the nat table, specifically the POSTROUTING chain, and the filter table’s FORWARD chain.

Nat Table Analysis:

Your iptables-save output for the nat table includes:

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [3497:641808]
:LIBVIRT_PRT - [0:0]
-A POSTROUTING -j LIBVIRT_PRT
-A POSTROUTING -o enp3s0 -j MASQUERADE
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
COMMIT

The critical rules here are:

-A POSTROUTING -o enp3s0 -j MASQUERADE: This rule masquerades traffic leaving the host via enp3s0. This is a common and correct way to provide internet access.
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE: This is the core libvirt NAT rule. It masquerades traffic originating from the 192.168.122.0/24 subnet (your VMs) when it’s not destined for within the same subnet.
The rules within LIBVIRT_PRT are also important for handling specific protocols and destinations correctly, ensuring that multicasts and broadcasts within the private subnet are not NATed.

The NAT rules appear to be correctly configured by libvirt for masquerading traffic. This implies the issue might not be with the NAT translation itself but rather with the forwarding of traffic or DNS resolution.

Filter Table Analysis:

Now let’s look at the filter table, specifically the FORWARD chain, as this controls whether packets are allowed to pass through the host from one network to another.

Your iptables-save output for the filter table includes:

*filter
:INPUT ACCEPT [21995:13427635]
:FORWARD ACCEPT [2:120]
:OUTPUT ACCEPT [20003:5430042]
:LIBVIRT_FWI - [0:0]
:LIBVIRT_FWO - [0:0]
:LIBVIRT_FWX - [0:0]
:LIBVIRT_INP - [0:0]
:LIBVIRT_OUT - [0:0]
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -d 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -d 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -d 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -d 68 -j ACCEPT
COMMIT

Let’s break down the relevant FORWARD rules:

-A FORWARD -j LIBVIRT_FWX: This jumps to a chain that allows traffic between virbr0 interfaces, which is unlikely to be the problem for internet access.
-A FORWARD -j LIBVIRT_FWI: This chain handles incoming traffic from guests to the host or other networks.
- -A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT: This rule allows established connections back to the guest’s network. This is correct for return traffic.
- -A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable: This is a potential problem. This rule seems to reject all incoming traffic on virbr0 that isn’t already established. If a new outgoing connection is initiated, the return packets might be rejected here before the NAT rule has a chance to properly associate them.
-A FORWARD -j LIBVIRT_FWO: This chain handles outgoing traffic from guests.
- -A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT: This allows traffic from the guest subnet originating from virbr0 to be accepted into the forward chain. This is good.
- -A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable: Similar to the previous reject rule, this might block necessary outgoing packets.

The most suspicious part in the filter table is the explicit REJECT rules that might be too aggressive and block legitimate forwarded traffic.

#### 5. Checking Host IP Forwarding (`/etc/sysctl.conf`)**

The output $ sudo cat /etc/sysctl.conf shows:

vm.swappiness=10
net.ipv4.ip_forward=1

This is excellent! net.ipv4.ip_forward=1 explicitly enables IP forwarding on the host, which is a prerequisite for NAT and routing traffic between interfaces. This setting is correctly applied.

#### 6. Host Routing Table (`route -n`)**

The host’s routing table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 enp3s0
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 enp3s0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 enp3s0
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 enp3s0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

This table shows:

A default route via 192.168.0.1 (your physical router) through enp3s0. This is correct for the host to access the internet.
The 192.168.122.0/24 network is directly connected via virbr0. This is also correct.

The host’s routing appears to be correctly configured.

#### 7. The Crucial Ping Test Discrepancy**

You mentioned: “The guest VM can ping the gateway 192.168.0.1 which is the router’s ip and gateway to internet but the VM can not ping the internet such as 8.8.8.8”.

This is a critical piece of information.

Guest pinging 192.168.0.1 (Host’s physical gateway): This proves that the guest can successfully send traffic to the host’s physical network interface (enp3s0) and that the host’s routing correctly directs this to the upstream router. It also implies that the NAT is working for this specific type of traffic (ICMP echo requests to the gateway).
Guest failing to ping 8.8.8.8: This indicates that while the guest can reach the physical gateway, the traffic destined for the broader internet is either not being forwarded correctly by the host, or a firewall is blocking it, or DNS resolution is failing.

Advanced Troubleshooting and Solutions

Based on our analysis, the most probable causes are either a misconfigured iptables rule that blocks outgoing traffic or, more commonly, a DNS resolution problem.

#### 8. Resolving DNS Issues for KVM/QEMU Guests**

The ability to ping an IP address like 8.8.8.8 confirms that the guest has basic IP connectivity to the internet. However, to browse websites, you need DNS to translate domain names into IP addresses.

**#### 8.1. Verifying DNS Server in Guest

While your ipconfig output doesn’t explicitly show DNS servers, the libvirt default configuration usually relies on the host’s DNS settings. A common reason for failure is that the guest is not receiving a DNS server from the libvirt DHCP server, or the DNS server it receives is not accessible.

How libvirt handles DNS:

Libvirt’s default network often configures its DNS forwarder to use the host’s /etc/resolv.conf.

Solution 1: Ensure Host DNS Resolution is Functional

First, let’s confirm that your host system can resolve domain names.

On the host, ping a domain name:
```
ping google.com
```
If this fails, you have a host-level DNS problem that needs to be addressed first. Ensure your /etc/resolv.conf on the host is correctly populated with working DNS servers (e.g., your router’s IP, Google DNS 8.8.8.8, Cloudflare DNS 1.1.1.1).
Check host’s /etc/resolv.conf:
```
cat /etc/resolv.conf
```
If it’s empty or contains invalid entries, you’ll need to fix it. For example, add:
```
nameserver 8.8.8.8
nameserver 1.1.1.1
```
Then, restart the NetworkManager or equivalent service on your host, or reboot the host.

Solution 2: Force Guest to Use Specific DNS Servers

If the host’s DNS is working but the guest isn’t getting it, we can explicitly configure the libvirt network to use specific DNS servers.

Edit the default network configuration:
```
sudo virsh net-edit default
```

Add a dns section within the <network> tag. If a dns section already exists, modify it. Add the forwarders section:

<network>
  <name>default</name>
  <uuid>f1eff8aa-73e7-4573-8d36-571a85714777</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:73:62:06'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
    <dns enable='yes'>
      <forwarder addr='8.8.8.8'/>
      <forwarder addr='1.1.1.1'/>
    </dns>
  </ip>
</network>

Explanation of the added lines:

<dns enable='yes'>: Enables DNS resolution for the network.
<forwarder addr='8.8.8.8'/>: Specifies Google’s public DNS server as a forwarder.
<forwarder addr='1.1.1.1'/>: Specifies Cloudflare’s public DNS server as an additional forwarder.

Save the changes and restart the network:

sudo virsh net-destroy default
sudo virsh net-start default
sudo virsh net-autostart default # Ensure it starts on boot

Inside the guest VM, renew its DHCP lease or reboot it.
- Windows: Open Command Prompt as Administrator and run:
```
ipconfig /release
ipconfig /renew
```
- Linux:
```
sudo dhclient -r
sudo dhclient
```
  or reboot the VM.
Test DNS resolution within the guest:
```
ping google.com
nslookup google.com
```

Solution 3: Configure Guest DNS Manually (Less Ideal)

If libvirt’s DHCP is persistently failing to provide DNS, you can manually set the DNS server within the guest. This is generally not recommended for long-term solutions as it bypasses the automation libvirt provides.

Windows: Go to Network Adapter settings, IPv4 properties, and manually set the DNS server to 192.168.122.1 (if libvirt’s internal DNS is working) or a public DNS like 8.8.8.8.
Linux: Edit /etc/resolv.conf or configure your network manager.

#### 9. Re-evaluating `iptables` Filter Rules**

While DNS is a common culprit, the presence of potentially overly restrictive REJECT rules in the filter table’s FORWARD chain is worth investigating. Libvirt’s default iptables rules are generally robust, but custom modifications or conflicts could interfere.

Let’s review the relevant parts of your filter table again:

*filter
...
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
...

These REJECT rules, particularly the one in LIBVIRT_FWI (which handles incoming traffic to the host from the guest’s perspective), might be prematurely blocking traffic. However, libvirt is designed to manage these chains and typically the ACCEPT rules for RELATED,ESTABLISHED connections handle the return traffic correctly.

Potential Issues with Filter Rules:

Order of Rules: The order of iptables rules is critical. A REJECT rule placed before an ACCEPT rule for established connections would cause problems.
Interference: Other services or manual iptables modifications might have altered the default libvirt rules.

Recommended Action for Filter Rules:

Trust Libvirt’s Defaults: If you haven’t made manual iptables changes, the rules generated by libvirt are usually correct. The most likely scenario is that the REJECT rules are not the primary cause if DNS or a basic forwarding block is the issue.
Temporarily Remove Aggressive REJECTs (for testing ONLY): Caution: This is a troubleshooting step and should be reverted. You could temporarily remove the problematic REJECT rules to see if connectivity resumes. This is a more advanced step and requires careful execution.
- Identify the exact rule number:
```
sudo iptables -t filter -L LIBVIRT_FWI --line-numbers
sudo iptables -t filter -L LIBVIRT_FWO --line-numbers
```
- Delete the rule (example, replace X with the actual line number):
```
sudo iptables -t filter -D LIBVIRT_FWI X
sudo iptables -t filter -D LIBVIRT_FWO Y
```
- Test connectivity.
- Re-add the rules: If the problem is resolved, you’ll need to understand why they were there. If not, re-add them immediately. Re-adding specific rules can be complex, so it’s often easier to restart the libvirt network to restore its default iptables rules.
Reload Libvirt Network Configuration: The safest way to ensure you have the correct iptables rules is to re-apply them via libvirt:
```
sudo virsh net-destroy default
sudo virsh net-start default
```
This will clear and re-establish the iptables rules managed by libvirt for the “default” network.

#### 10. IP Forwarding on the Host Revisited**

While you confirmed net.ipv4.ip_forward=1 is set, sometimes changes aren’t applied correctly, or a reboot is needed for them to fully take effect.

Verify if forwarding is actually enabled:
```
sysctl net.ipv4.ip_forward
```
The output should be net.ipv4.ip_forward = 1.
If it’s 0, enable it temporarily:
```
sudo sysctl -w net.ipv4.ip_forward=1
```
And ensure it’s persistent by having net.ipv4.ip_forward=1 in /etc/sysctl.conf and applying it with sudo sysctl -p.

#### 11. Host Interface Configuration (`enp3s0`)**

Ensure your host’s primary network interface (enp3s0 in your case) has a valid IP address and is functioning correctly. The guest’s ability to ping the router (192.168.0.1) suggests this is working, but it’s a fundamental check.

Summary of Solutions and Best Practices

Based on the provided information and typical KVM/QEMU networking issues, the most probable cause for your guest’s lack of internet connectivity, despite being able to ping the gateway, is DNS resolution failure.

Here’s a prioritized list of actions:

Verify Host DNS: Ensure your host can resolve domain names (ping google.com). Fix /etc/resolv.conf if necessary.
Configure Libvirt DNS Forwarders: Edit the default network (sudo virsh net-edit default) to include DNS forwarders, pointing to public DNS servers like 8.8.8.8 and 1.1.1.1. Restart the default network (sudo virsh net-destroy default && sudo virsh net-start default).
Renew Guest DHCP Lease: After changing the libvirt network, renew the IP address on the guest VM.
Test DNS in Guest: Use ping google.com or nslookup google.com from within the guest.

If DNS is not the issue, then:

Re-apply Libvirt Network Rules: Destroy and start the default network (sudo virsh net-destroy default && sudo virsh net-start default) to reset iptables to libvirt’s managed state. This is the most reliable way to ensure the NAT and filter rules are correctly applied.
Verify IP Forwarding: Double-check that net.ipv4.ip_forward is 1 on the host.

By meticulously following these steps, we can systematically diagnose and resolve the internet connectivity problems for your KVM/QEMU guests utilizing the default NAT network. At revWhiteShadow, we aim to provide the clarity and detail necessary to empower you in managing your virtualized infrastructure effectively. We are confident that by addressing these key areas, particularly the DNS configuration, your virtual machines will regain full internet access.

KVM/QEMU libvirt Network ‘default’ NAT Configuration - Guest cannot connect to Internet no VPN

Mastering KVM/QEMU Libvirt Network “default” NAT: Solving Guest Internet Connectivity Issues #

Understanding the KVM/QEMU Libvirt “default” NAT Network Architecture #

Diagnosing the Root Cause: A Step-by-Step Approach #

**#### 1. Verifying Network Configuration (virsh net-dumpxml default) #

**#### 2. Examining Host Network Interface (ip address show dev virbr0) #

**#### 3. Guest IP Configuration (ipconfig in Windows) #

**#### 4. Analyzing iptables Rules (sudo iptables-save) #

**#### 5. Checking Host IP Forwarding (/etc/sysctl.conf) #

**#### 6. Host Routing Table (route -n) #

**#### 7. The Crucial Ping Test Discrepancy #

Advanced Troubleshooting and Solutions #

**#### 8. Resolving DNS Issues for KVM/QEMU Guests #

**#### 9. Re-evaluating iptables Filter Rules #

**#### 10. IP Forwarding on the Host Revisited #

**#### 11. Host Interface Configuration (enp3s0) #

Summary of Solutions and Best Practices #