Troubleshooting systemd-resolved: Resolving DNS Lookup Failures on Debian

Welcome to the revWhiteShadow blog, your go-to resource for in-depth Linux troubleshooting and system administration tips. We’re revWhiteShadow and kts, and in this article, we delve into a frustrating issue that can arise after installing systemd-resolved on Debian-based systems, specifically when attempting to get Home Assistant running: the dreaded DNS lookup failure. We’ll explore potential causes, solutions, and preventative measures to ensure a smooth and functional network resolution setup. We’ll cover everything from basic configuration checks to more advanced debugging techniques, helping you diagnose and resolve DNS problems effectively.

Understanding the systemd-resolved DNS Conundrum

The systemd-resolved service is a system service that provides network name resolution to local applications. It acts as a local DNS stub resolver, caching DNS records and forwarding queries to configured DNS servers. It’s designed to simplify DNS configuration and improve performance. However, misconfiguration or conflicts with existing network settings can lead to DNS resolution failures, leaving your system unable to translate domain names into IP addresses. This is exactly what happened when trying to install Home Assistant supervised on a Debian system, as we discovered.

The Problem: DNS Resolution Fails After Installing systemd-resolved

As observed, after installing systemd-resolved on a Debian system, the ability to resolve domain names, like github.com, ceased to function. While pinging IP addresses worked flawlessly, attempting to wget or access websites by their domain names resulted in “Temporary failure in name resolution” errors. This indicates a problem with how the system is translating domain names into their corresponding IP addresses.

Here is a summary of the issue:

  • Symptom: Inability to resolve domain names after installing systemd-resolved.
  • Environment: Debian-based system (specifically, Orange Pi Zero).
  • Error Message: “Temporary failure in name resolution”.
  • Successful Operation: Pinging IP addresses still functions correctly.

Initial Observations and Environment Details

The problem was initially encountered while following the Home-Assistant supervised installer instructions on an Orange Pi Zero running Debian. Notably, this issue did not occur when using an Armbian image. Armbian is not a supported OS for this setup, which lead to the need to troubleshoot the issue in Debian. The fact that the issue only manifested on Debian suggests a possible incompatibility or configuration difference between the two operating systems in how they handle DNS resolution with systemd-resolved.

The network setup consisted of a TP-Link home router with DHCP address reservation for the device’s MAC address. The device was on the same subnet as other devices, and the router did not specify a different DNS server specifically for the device.

Diagnosing the Root Cause of DNS Failure

Before diving into solutions, we need to understand why systemd-resolved might be causing DNS resolution to fail. Several factors can contribute to this issue:

1. /etc/resolv.conf Configuration:

The /etc/resolv.conf file is traditionally used to specify DNS servers. systemd-resolved can manage this file in different ways, depending on the system configuration.

  • Scenario 1: systemd-resolved creates a symbolic link from /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf. This file contains nameserver 127.0.0.53, indicating that DNS queries are being directed to the local systemd-resolved stub resolver.
  • Scenario 2: A static /etc/resolv.conf file is used, overriding systemd-resolved’s default behavior.

The critical point is that the DNS servers specified in /etc/resolv.conf (either directly or indirectly through the stub resolver) must be valid and reachable for DNS resolution to work.

Here’s what was seen in testing:

  • Before systemd-resolved: /etc/resolv.conf contained nameserver 192.168.0.1 (likely the router) and nameserver 0.0.0.0.
  • After First Install: /etc/resolv.conf became a symlink to /run/systemd/resolve/stub-resolv.conf, containing nameserver 127.0.0.53.
  • After Uninstall and Re-install: /etc/resolv.conf reverted to its original state (nameserver 192.168.0.1 and nameserver 0.0.0.0).

2. /etc/nsswitch.conf Configuration

The /etc/nsswitch.conf file specifies the order in which the system should consult different sources for name resolution (e.g., files, DNS, etc.). If systemd-resolved is not properly integrated into this file, DNS lookups may not be performed correctly.

Specifically, the hosts: line in /etc/nsswitch.conf controls how hostnames are resolved. After installing systemd-resolved, the hosts: line was modified to include resolve:

  • Before systemd-resolved: hosts: files dns
  • After systemd-resolved: hosts: files resolve [!UNAVAIL=return] dns myhostname

The resolve option instructs the system to use systemd-resolved for hostname resolution. The [!UNAVAIL=return] part means that if systemd-resolved is unavailable, the system should immediately proceed to the next source (in this case, dns).

3. systemd-resolved Service Status

The systemd-resolved service must be running and functioning correctly for DNS resolution to work. If the service is stopped, failing to start, or encountering errors, DNS lookups will likely fail. The output of systemctl status systemd-resolved is crucial for diagnosing service-related issues.

4. Network Connectivity

Basic network connectivity is a prerequisite for DNS resolution. If the system cannot reach the configured DNS servers (e.g., due to firewall rules, routing issues, or network outages), DNS lookups will fail. Being able to ping IP addresses but not domain names suggests that network connectivity is generally working, but there’s a problem specifically with DNS resolution.

Troubleshooting and Resolving DNS Lookup Failures

Based on the potential causes outlined above, here’s a step-by-step approach to troubleshooting and resolving DNS lookup failures after installing systemd-resolved:

Step 1: Verify systemd-resolved Service Status

The first step is to check the status of the systemd-resolved service:

systemctl status systemd-resolved
  • If the service is not running (inactive): Start the service:
sudo systemctl start systemd-resolved
  • If the service is failing to start: Check the journal for errors:
journalctl -u systemd-resolved

Look for any error messages that might indicate the cause of the failure. Common issues include configuration errors, conflicts with other services, or missing dependencies.

  • If the service is running but encountering errors: The output of systemctl status may show error messages or warnings. Investigate these messages to identify the root cause of the problem.

In the original scenario, the service was running, but resolvectl status failed with “Connection timed out,” suggesting a problem with the communication between the resolvectl utility and the systemd-resolved service.

Step 2: Inspect /etc/resolv.conf

Examine the contents of /etc/resolv.conf to determine how it’s configured.

  • If it’s a symlink to /run/systemd/resolve/stub-resolv.conf: This is the default configuration when systemd-resolved is managing DNS resolution. Ensure that the systemd-resolved service is properly configured and that the DNS servers it’s using are reachable.
  • If it’s a static file: Check that the nameserver entries are correct and point to valid DNS servers. You can try using public DNS servers like Google’s (8.8.8.8 and 8.8.4.4) or Cloudflare’s (1.1.1.1 and 1.0.0.1) to rule out issues with your ISP’s DNS servers.
cat /etc/resolv.conf

If /etc/resolv.conf contains nameserver 127.0.0.53, you can use resolvectl status to check the upstream DNS servers being used by systemd-resolved. This command provides detailed information about the current DNS configuration.

Step 3: Examine /etc/nsswitch.conf

Verify that /etc/nsswitch.conf is configured to use systemd-resolved for hostname resolution:

cat /etc/nsswitch.conf

The hosts: line should include resolve, typically in the following format:

hosts: files resolve [!UNAVAIL=return] dns myhostname

If resolve is missing, add it to the hosts: line.

Step 4: Use resolvectl for DNS Queries

The resolvectl command can be used to query DNS servers directly through systemd-resolved. This is a useful tool for testing DNS resolution and diagnosing problems.

  • Query a specific domain:
resolvectl query github.com

This command will attempt to resolve github.com using systemd-resolved. If the query fails, the output will provide error messages that can help you identify the problem.

  • Check the status of a specific interface:
resolvectl status eth0

Replace eth0 with the name of your network interface. This command will show the DNS servers configured for that interface and their status.

Step 5: Restart systemd-resolved and Network Services

After making changes to configuration files, restart systemd-resolved and related network services to apply the changes:

sudo systemctl restart systemd-resolved
sudo systemctl restart networking # may not be applicable on systemd based systems

Step 6: Consider DNS Server Reachability and Firewall Rules

Ensure your system can reach the DNS servers it’s configured to use. Use ping or traceroute to test connectivity:

ping 8.8.8.8
traceroute 8.8.8.8

If you cannot reach the DNS servers, check your firewall rules and routing configuration. Make sure that DNS traffic (port 53) is allowed.

Step 7: The Curious Case of Reinstallation

In the original problem, reinstalling systemd-resolved seemed to fix the issue. This is unexpected, but here’s a possible explanation:

  • Configuration Reset: The reinstallation process might have reset some configuration files or settings that were causing the problem.
  • Package Issue: It’s possible that the initial installation was somehow corrupted or incomplete, and the reinstallation corrected this.
  • Idempotency: Installation and uninstallation processes, while being intended to do the same thing when repeated do not guarantee the same outcome. This is often due to external factors such as package updates in the middle of the process.

The fact that /etc/resolv.conf was not modified after the second installation suggests that the reinstallation process might have bypassed the default behavior of creating a symlink to /run/systemd/resolve/stub-resolv.conf. This could have inadvertently resolved the issue by reverting to the original DNS configuration.

Long-Term Solutions and Best Practices

While reinstalling systemd-resolved might provide a temporary fix, it’s essential to understand the underlying cause of the problem and implement a more sustainable solution.

1. Ensure Correct DNS Server Configuration

The most important step is to ensure that your system is configured to use valid and reachable DNS servers. This can be done in several ways:

  • DHCP: If your router provides DHCP, it will typically also provide DNS server addresses. This is the simplest and most common approach.
  • Static Configuration: You can manually configure DNS servers in /etc/resolv.conf or through network configuration tools. This gives you more control but requires more manual configuration.
  • systemd-resolved: When using systemd-resolved, you can configure DNS servers globally or for specific interfaces.

2. Properly Integrate systemd-resolved

Make sure that /etc/nsswitch.conf is correctly configured to use systemd-resolved for hostname resolution. This ensures that the system consults systemd-resolved when performing DNS lookups.

3. Monitor systemd-resolved Service

Regularly monitor the status of the systemd-resolved service to ensure that it’s running correctly and not encountering any errors. Use systemctl status systemd-resolved and journalctl -u systemd-resolved to check for issues.

4. Keep Your System Up-to-Date

Ensure that your system is running the latest versions of systemd, systemd-resolved, and other relevant packages. Updates often include bug fixes and performance improvements that can resolve DNS-related issues.

Conclusion: Mastering DNS Resolution with systemd-resolved

systemd-resolved is a powerful tool for managing DNS resolution, but it can also be a source of frustration if not configured correctly. By understanding the potential causes of DNS lookup failures and following the troubleshooting steps outlined in this article, you can effectively diagnose and resolve DNS problems on your Debian system. Remember to pay close attention to the /etc/resolv.conf and /etc/nsswitch.conf files, monitor the systemd-resolved service status, and ensure that your system can reach the configured DNS servers.

We, at revWhiteShadow, hope this comprehensive guide has equipped you with the knowledge and tools you need to confidently tackle DNS resolution issues with systemd-resolved. Happy troubleshooting!