Troubleshooting: sshd Service Fails to Start on Arch Linux Boot

When embarking on the journey of setting up a new Arch Linux system, the ability to remotely access your machine via SSH is often a primary goal. The OpenSSH server, or sshd, is the cornerstone of this functionality. However, encountering a situation where the sshd service fails to automatically start upon system boot, despite manually initiating it with success, can be a perplexing and frustrating experience. At revWhiteShadow, your trusted personal blog for insightful tech solutions, we understand the critical nature of a reliable remote connection. This comprehensive guide is meticulously crafted to help you diagnose and resolve the persistent issue of the sshd service not starting automatically when your Arch Linux system boots. We will delve deep into the potential causes and provide actionable, detailed steps to ensure your sshd daemon is always ready to accept your remote connections from the moment your system comes online.

Understanding the `sshd` Service and Systemd

Before we dive into troubleshooting, it’s essential to grasp how services are managed in Arch Linux. Arch Linux, like many modern Linux distributions, utilizes systemd as its primary system and service manager. Systemd is responsible for initializing the system, managing processes, and orchestrating the startup of various services. Each service, including sshd, is defined by a service unit file, typically located in /usr/lib/systemd/system/ or /etc/systemd/system/.

The sshd.service unit file contains directives that tell systemd how to start, stop, and manage the OpenSSH server daemon. When you enable a service using systemctl enable sshd.service, systemd creates a symbolic link, usually in /etc/systemd/system/multi-user.target.wants/, pointing to the service unit file. This link informs systemd that the service should be started when the system reaches the multi-user.target state, which is the standard runlevel for a non-graphical multi-user system.

The fact that you can successfully start sshd manually using sudo /usr/bin/sshd is a crucial piece of information. This indicates that the OpenSSH server is installed correctly, the sshd executable is present and functional, and your system’s fundamental configuration allows the daemon to run. The problem, therefore, lies not with the sshd binary itself, but with the systemd’s ability to orchestrate its startup process automatically.

Diagnosing the `sshd` Service Failure: A Deeper Dive

The output from sudo systemctl status sshd.service provides invaluable clues. Let’s dissect the common reasons for the Active: failed (Result: exit-code) status when sshd is intended to start on boot:

Incorrect sshd.service Unit File Configuration: The service unit file might have errors or missing crucial directives that prevent systemd from launching it correctly.
Dependency Issues: sshd might depend on other services or system states that are not yet ready when its startup is attempted.
Permissions or Ownership Problems: Critical files or directories used by sshd might have incorrect permissions, preventing the daemon from accessing them.
Network Configuration: While sshd can start without a network, it expects certain network interfaces to be up and configured. Issues here can sometimes cause startup failures.
Port Conflicts: Although less common for default SSH port 22, if another service is already bound to the SSH port, sshd will fail to start.
Configuration File Errors (sshd_config): Syntax errors or invalid directives within /etc/ssh/sshd_config can lead to the sshd daemon failing to initialize.
Systemd Version or Specific Arch Linux Quirks: Occasionally, specific versions of systemd or unique configurations within Arch Linux might introduce subtle startup behaviors.

We will systematically address each of these potential problem areas.

Initial Verification: Ensuring the Basics Are Covered

Before we delve into complex diagnostics, let’s re-verify the fundamental steps and configurations:

1. Confirming `sshd` Installation and Service File Existence

Ensure that the openssh package is indeed installed and that the sshd.service file exists in the expected locations.

Check Package Installation:
```
pacman -Q openssh
```
If this command returns the package name and version, openssh is installed. If not, install it:
```
sudo pacman -S openssh
```
Verify Service File Location: sshd should have a default service file provided by the openssh package. The primary location for this is /usr/lib/systemd/system/sshd.service.
```
ls -l /usr/lib/systemd/system/sshd.service
```
You should see output indicating the file exists.

2. Enabling and Starting the `sshd` Service Manually (Again)

We already know manual start works, but let’s ensure the service is properly enabled for startup.

Enable the Service:
```
sudo systemctl enable sshd.service
```
This command creates the necessary symbolic link for automatic startup.
Check the Status Again:
```
sudo systemctl status sshd.service
```
Even if it shows failed, observe the output for any new error messages or changes.

3. Verifying Network Interface Status

sshd needs a network interface to bind to. While it can start without an active internet connection, the interface itself should be brought up.

Check Network Manager or systemd-networkd: If you are using NetworkManager, check its status:
```
systemctl status NetworkManager.service
```
If you are using systemd-networkd, check its status:
```
systemctl status systemd-networkd.service
```
Ensure your primary network interface (e.g., eth0, enpXsY) is active and has an IP address. You can check this with:
```
ip a
```

Advanced Troubleshooting: Pinpointing the Root Cause

If the initial checks don’t resolve the issue, we must delve deeper into potential configuration problems and system interactions.

4. Examining the `sshd.service` Unit File for Customizations or Errors

While Arch Linux provides a default sshd.service file, customizations in /etc/systemd/system/ can override it. It’s crucial to ensure these customizations are correct or to revert to the default if they are causing issues.

Check for Overrides in /etc/systemd/system/:
```
ls -l /etc/systemd/system/sshd.service
```
If this command shows the file exists in /etc/systemd/system/, it means it’s overriding the one in /usr/lib/systemd/system/.
Compare with the Default: It’s a good practice to compare your custom unit file (if any) with the one provided by the openssh package.
```
sudo cat /usr/lib/systemd/system/sshd.service
```
Look for any discrepancies, particularly in the ExecStart line, Restart directives, and WantedBy clauses. The ExecStart line should typically look something like:
```
ExecStart=/usr/bin/sshd -D
```
The Restart directive is also important; Restart=on-failure or Restart=always are common for daemons.
Reverting to Default (If Necessary): If you suspect your custom unit file is the culprit, you can remove it to allow systemd to use the default from /usr/lib/systemd/system/.
```
sudo rm /etc/systemd/system/sshd.service
sudo systemctl daemon-reload
sudo systemctl restart sshd.service
```
Then, re-enable and try booting again:
```
sudo systemctl enable sshd.service
sudo reboot
```

5. Investigating `sshd_config` for Syntax Errors and Critical Directives

The sshd_config file (/etc/ssh/sshd_config) contains all the configuration options for the OpenSSH server. Errors here are a very common cause of startup failures.

Check for Syntax Errors: sshd itself can validate its configuration file.
```
sudo sshd -t
```
If there are any syntax errors, this command will report them, usually indicating the line number and the nature of the error. Fix any reported errors immediately.
Common sshd_config Issues:
- ListenAddress: If ListenAddress is uncommented and set to an IP address that is not available on your system, sshd might fail. Ensure it’s either commented out (to listen on all interfaces) or set to a valid IP address.
- Port: While unlikely for the default port 22, ensure the specified port is not already in use by another service. You can check this with sudo ss -tulnp | grep :22.
- UsePrivilegeSeparation: This directive is crucial for security. Ensure it’s set to yes. If you have manually changed it to no and are experiencing issues, revert it to yes. The default value is usually yes.
- PidFile: Ensure the directory where the PID file is supposed to be written has correct permissions. However, default configurations usually handle this.
Backing Up and Resetting sshd_config: If you’ve made many changes to sshd_config and are unsure, it’s safest to back it up and revert to a known good state.
```
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup_$(date +%Y%m%d_%H%M%S)
```
Then, you can try using the default configuration provided by the openssh package, although Arch typically doesn’t overwrite user-modified config files. A safer approach is to carefully review your changes and fix them based on sshd -t output.

6. Analyzing Systemd Journal Logs for Detailed Error Information

The systemd journal is the central logging system for your Arch Linux installation. It provides detailed information about service startup and any errors encountered. This is arguably the most important tool for diagnosing startup failures.

View Logs for sshd.service:
```
sudo journalctl -u sshd.service
```
This command displays all logs related to the sshd.service. Look for lines marked with error, failed, exited with status, or other indicators of problems around the time of boot or when you attempted to start the service.
View Logs from Boot Time: To see what happened during the last boot:
```
sudo journalctl -b -u sshd.service
```
The -b flag shows logs from the current boot.
Follow Logs in Real-Time: To monitor logs as they are generated (useful if you plan to restart the service):
```
sudo journalctl -f -u sshd.service
```
What to Look For in the Logs:
- “Permission denied”: This indicates a file or directory permission issue.
- “Address already in use”: Means another process is using the port sshd is trying to bind to.
- “Bad configuration option”: Points to an error in sshd_config.
- “Could not resolve hostname”: If sshd tries to resolve a hostname during startup and it fails.
- “Failed to bind to…”: Often related to network interface or port binding issues.

7. Checking File Permissions and Ownership

Incorrect permissions on critical directories and files used by sshd can prevent it from starting.

SSH Directory Permissions: The /etc/ssh/ directory and its contents are crucial.
```
ls -ld /etc/ssh/
ls -l /etc/ssh/sshd_config
```
The /etc/ssh/ directory should typically be owned by root:root with permissions like 755 or 700. The sshd_config file should be owned by root:root and be readable only by root (600 or 644).
SSH Host Key Permissions: The host keys generated by ssh-keygen are essential for sshd operation. They are typically located in /etc/ssh/.
```
ls -l /etc/ssh/ssh_host_*_key
ls -l /etc/ssh/ssh_host_*_key.pub
```
The private host key files (e.g., ssh_host_rsa_key) should be owned by root:root and have strict permissions, usually 600, to prevent unauthorized access. The public key files can be more permissive, typically 644.
Fixing Permissions: If permissions are incorrect, use chmod and chown to correct them. For example:
```
sudo chown root:root /etc/ssh/ssh_host_rsa_key
sudo chmod 600 /etc/ssh/ssh_host_rsa_key
```
Repeat for other host key files as needed.
Re-generating Host Keys: If you suspect the host keys themselves are corrupted or the permissions are too complex to fix, you can regenerate them. Warning: This will change your server’s SSH host key, and clients will receive a warning about a changed host key upon their next connection.
```
sudo rm /etc/ssh/ssh_host_*_key
sudo rm /etc/ssh/ssh_host_*_key.pub
sudo ssh-keygen -A
```
Then, reload systemd and restart sshd:
```
sudo systemctl daemon-reload
sudo systemctl restart sshd.service
```
And re-enable it for boot:
```
sudo systemctl enable sshd.service
sudo reboot
```

8. Checking for Network Interface Dependencies and Ordering

sshd needs a network interface to listen on. If the network interface isn’t fully initialized before sshd attempts to start, it can fail. systemd manages these dependencies.

Understanding After= and Wants= Directives: Examine the sshd.service unit file (from step 4) for After= and Wants= directives. These specify dependencies. Typically, sshd.service will have directives like:
```
After=network.target
Wants=network.target
```
Or more specific targets related to network configuration.
Ensuring Network Targets are Met: The network.target is a generic target that signifies network availability. Arch Linux’s network management solutions (like NetworkManager or systemd-networkd) should activate this target. If your network setup is unconventional or has delayed startup, sshd might be starting too early.
Adjusting Network Dependencies (Advanced): If you are absolutely certain that network startup is the bottleneck, you could, in theory, add specific After= directives to your custom sshd.service unit file to wait for a particular network interface to be brought up. However, this is rarely necessary with standard Arch Linux network configurations and can introduce its own complexities. It’s usually better to ensure your primary network manager is functioning correctly and enabled for boot.

9. Verifying PAM (Pluggable Authentication Modules) Configuration

While less common for the sshd service itself failing to start, issues with PAM configuration could, in theory, affect how the sshd daemon initializes, especially if it relies on specific PAM modules during its startup phase.

PAM Configuration Files: The relevant PAM configuration for sshd is typically found in /etc/pam.d/sshd.
```
sudo cat /etc/pam.d/sshd
```
Again, syntax errors or incorrect module calls here could lead to unexpected behavior. However, the sshd -t command usually catches most critical configuration errors, and PAM errors are more likely to manifest during user login rather than service startup.

10. Considering Potential Conflicts with Other Services

Although unlikely for standard openssh installations, it’s worth considering if any other recently installed or configured services might interfere with sshd.

Firewall Rules: While a firewall typically blocks connections rather than preventing the service from starting, overly aggressive or misconfigured firewall rules could potentially interfere with the binding process. Ensure your firewall (e.g., iptables, nftables, ufw) is configured correctly to allow traffic on port 22 if you are using it.
SELinux/AppArmor: Arch Linux does not typically enable SELinux or AppArmor by default. If you have manually enabled them, their policies could be too restrictive.

Final Steps and Best Practices

After implementing the troubleshooting steps, it’s crucial to perform a clean restart and verification.

1. Clean Re-enable and Reboot

Once you believe you have identified and corrected the issue, perform these steps:

Reload Systemd:
```
sudo systemctl daemon-reload
```
Restart the sshd Service:
```
sudo systemctl restart sshd.service
```
Check Status:
```
sudo systemctl status sshd.service
```
Verify it’s now active (running).
Enable for Boot:
```
sudo systemctl enable sshd.service
```
Reboot the System:
```
sudo reboot
```
Verify After Reboot: After the system has booted up, log in and check the status of the sshd service:
```
sudo systemctl status sshd.service
```
And then try to connect from another machine:
```
ssh your_username@your_arch_linux_ip_address
```

2. Maintaining a Healthy `sshd` Configuration

Regularly update your system: Keep your Arch Linux system updated with sudo pacman -Syu. Updates often include newer versions of openssh with bug fixes.
Test changes carefully: When modifying configuration files like sshd_config, always test the syntax (sudo sshd -t) and then restart the service (sudo systemctl restart sshd.service).
Keep backups: Maintain backups of critical configuration files, especially /etc/ssh/sshd_config.

By systematically working through these detailed steps, you should be able to diagnose and resolve the issue of your sshd service failing to start on boot in Arch Linux. The key lies in careful observation of error messages, understanding the role of systemd, and meticulously checking configuration files and permissions. We are confident that this comprehensive guide from revWhiteShadow will empower you to restore reliable SSH access to your Arch Linux system.

sshd service doesn’t start when booting Arch Linux

Troubleshooting: sshd Service Fails to Start on Arch Linux Boot #

Understanding the sshd Service and Systemd #

Diagnosing the sshd Service Failure: A Deeper Dive #

Initial Verification: Ensuring the Basics Are Covered #

1. Confirming sshd Installation and Service File Existence #

2. Enabling and Starting the sshd Service Manually (Again) #

3. Verifying Network Interface Status #

Advanced Troubleshooting: Pinpointing the Root Cause #

4. Examining the sshd.service Unit File for Customizations or Errors #

5. Investigating sshd_config for Syntax Errors and Critical Directives #

6. Analyzing Systemd Journal Logs for Detailed Error Information #

7. Checking File Permissions and Ownership #

8. Checking for Network Interface Dependencies and Ordering #

9. Verifying PAM (Pluggable Authentication Modules) Configuration #

10. Considering Potential Conflicts with Other Services #

Final Steps and Best Practices #

1. Clean Re-enable and Reboot #

2. Maintaining a Healthy sshd Configuration #