Mastering Bind Mounts with systemd: Binding to Device Paths Instead of Specified Directories #

This article delves into a common yet often perplexing issue encountered when configuring systemd mount units for bind mounts: the system’s tendency to incorrectly interpret or bind mount targets to device paths rather than the explicitly defined source directories. We will thoroughly explore the intricacies of this behavior, providing a detailed, actionable guide for achieving reliable and predictable bind mounts in your system. Our objective is to furnish you with the knowledge and techniques necessary to ensure your systemd mount units function precisely as intended, every time, even across system reboots.

At revWhiteShadow, we understand the critical role of precise system configuration in maintaining a stable and efficient operating environment. Bind mounts, a powerful feature of the Linux kernel, allow us to make a directory or file accessible at another location. When combined with systemd, they offer a robust mechanism for managing these mounts. However, as the scenario presented illustrates, misconfigurations can lead to unexpected behavior, particularly during the boot process.

Understanding the Discrepancy: Device Paths vs. Directory Paths in Bind Mounts #

The core of the issue lies in how systemd and the underlying mount utilities interpret the What= directive in a .mount unit file. While we intend to specify a directory path, such as /volume1/nix, the system sometimes resolves this to an underlying device path, like /dev/md0, particularly if /volume1/nix itself is an existing mount point for a filesystem.

This behavior is not necessarily a flaw but rather a consequence of the system’s internal logic for managing storage and mount points. When a directory is already associated with a block device, the mount command, and by extension systemd when it invokes it, might prioritize the direct device association over the abstract directory path provided in the configuration. This can lead to the observed discrepancy where a manual systemctl restart nix.mount corrects the mount, but a system reboot does not, because the initial boot sequence might not have fully established the intended environment for the directory-based bind mount.

The systemd Mount Unit: A Detailed Examination #

Let us dissect the provided nix.mount unit file to understand its components and potential areas of misinterpretation:

[Mount]
What=/volume1/nix
Where=/nix
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

• [Mount] Section: This section contains the core directives for the mount operation.

  • What=: This directive specifies the source of the mount. In our case, /volume1/nix.
  • Where=: This directive specifies the mount point, the destination where the source will be accessible. Here, it’s /nix.
  • Type=: This specifies the filesystem type. For a bind mount, none is generally used because it’s not a traditional filesystem type but rather a way of re-mounting an existing filesystem.
  • Options=: This is where bind is specified, indicating that this is a bind mount operation.

• [Install] Section: This section dictates how the unit is enabled to start automatically.

  • WantedBy=local-fs.target: This directive ensures that the nix.mount unit is started after the local file systems have been mounted. This is a crucial part of the boot process, but it also highlights where timing issues might arise.

The issue might stem from the fact that /volume1/nix itself could be a mount point for a larger filesystem (e.g., from a partition or RAID array like /dev/md0). During the early stages of the boot process, systemd might be processing mount units before the underlying device and its filesystem are fully prepared or before the system resolves the directory path to its underlying device representation in a way that aligns with the bind mount’s intention.

The Root Cause: Race Conditions and Mount Point Dependencies #

The behavior observed – working on restart but not on reboot – strongly suggests a race condition. This occurs when two or more processes or threads are accessing a shared resource, and the final outcome depends on the particular order in which they execute. In the context of systemd and mounts, this can happen if the target directory (/nix) or the source directory (/volume1/nix) is not in the state expected by the nix.mount unit at the precise moment it’s being processed during the boot sequence.

Specifically, if /volume1 is itself a mount point that depends on a device (like /dev/md0), and that device or its filesystem isn’t fully ready when nix.mount attempts to bind /volume1/nix to /nix, systemd might fall back to a more primitive device-based mount interpretation. This is compounded if /nix already exists as an empty directory.

The manual systemctl restart nix.mount works because at that point, the system is already running, all necessary devices are mounted, and the environment is stable. systemd can then reliably interpret /volume1/nix as the intended source for the bind mount.

The Correct Approach: Utilizing systemd-mount and .mount Units with Enhanced Specificity #

To reliably achieve the desired bind mount, we need to ensure systemd correctly interprets the source directory and its dependencies. This often involves a more robust configuration that accounts for the underlying filesystem and device. While a direct .mount unit is the standard for persistent mounts, for bind mounts that rely on other mount points, we can leverage systemd-mount or refine the .mount unit.

Method 1: Refining the .mount Unit for Robustness #

The most direct way to fix this is to ensure the .mount unit is processed in a context where the source filesystem is guaranteed to be available. While local-fs.target is generally appropriate, we can sometimes gain more control by specifying dependencies more explicitly.

Understanding What= and Device Paths

The fundamental issue is that the What= field in a .mount unit is generally expected to be a block device (like /dev/sda1) or a UUID/LABEL referring to a filesystem. When you specify a directory that is itself a mount point, systemd might try to resolve this to the underlying device.

To circumvent this, we need to tell systemd that we are bind-mounting a directory, not directly a device. This is achieved by using Type=none and Options=bind, which you have already done. The trick lies in ensuring the source path is correctly understood.

One effective strategy is to ensure the parent directory of your source (/volume1 in this case) is mounted and available before the bind mount unit is activated.

Ensuring Parent Directory Availability #

If /volume1 is mounted via another systemd unit (e.g., a separate /etc/systemd/system/volume1.mount file), the nix.mount unit needs to depend on it.

Let’s assume you have a volume1.mount unit defined as follows:

# /etc/systemd/system/volume1.mount
[Unit]
Description=Mount for Volume 1

[Mount]
What=/dev/md0
Where=/volume1
Type=ext4  # Or your actual filesystem type
Options=defaults

[Install]
WantedBy=local-fs.target

In this scenario, your nix.mount unit should explicitly depend on volume1.mount:

# /etc/systemd/system/nix.mount
[Unit]
Description=Nix Bind Mount to Volume1
Requires=volume1.mount  # Guarantees volume1.mount is active
After=volume1.mount     # Ensures nix.mount starts after volume1.mount

[Mount]
What=/volume1/nix
Where=/nix
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

By adding Requires=volume1.mount and After=volume1.mount, we are instructing systemd to activate volume1.mount before it attempts to process nix.mount. This significantly reduces the chance of a race condition where the source directory /volume1/nix might not be properly established or accessible.

Detailed Breakdown of Requires and After:

• Requires=volume1.mount: This is a strong dependency. If volume1.mount fails to activate, nix.mount will also be deactivated. This ensures that the necessary parent mount is in place.
• After=volume1.mount: This directive establishes an ordering. nix.mount will only be considered for activation after volume1.mount has been successfully activated.

After making these changes to your nix.mount file, remember to:

1. Reload the systemd daemon: sudo systemctl daemon-reload
2. Restart the service: sudo systemctl restart nix.mount
3. Verify the mount: mount | grep nix and ls -l /nix
4. Reboot your system and verify again.

Alternative Source Path Specification (Less Common for Bind Mounts) #

In some niche cases, explicitly referencing the device path might be considered, though this moves away from the spirit of bind-mounting a directory. For example, if you know /volume1 is mounted from /dev/md0, you might be tempted to try:

# POTENTIALLY INCORRECT OR LESS ROBUST APPROACH
[Mount]
What=/dev/md0:/nix  # This syntax is not standard for systemd .mount files for bind mounts
# Instead, what is intended is that systemd resolves /volume1/nix to a valid path on /dev/md0

However, the standard and correct way to perform a bind mount using systemd .mount units is to specify the directory path as you have done. The challenge is ensuring the path is resolvable as intended.

Method 2: Leveraging systemd-mount for Temporary or Dynamic Mounts #

While .mount units are for persistent configurations managed by systemd, systemd-mount is a command-line utility that can be used to mount filesystems, and it can also be integrated into systemd services or timers. For dynamic or more complex scenarios, this might be an alternative, though for a standard boot-time mount, a .mount unit is usually preferred.

The core idea is that systemd-mount can interpret options more flexibly. However, for your specific goal of a persistent, boot-time bind mount, sticking with the .mount unit and ensuring proper dependencies is the recommended and more idiomatic systemd approach.

Let’s focus on refining the .mount unit as it’s the direct answer to your problem.

Pre-Mount Hooks or Scripts #

If even adding Requires and After doesn’t resolve the issue, it implies that /volume1 itself might be experiencing delays in its own mounting process. In such scenarios, one might consider using systemd’s powerful but more advanced features like .preset files for ordering or even custom .service units that perform the mount using the mount command with specific options, then inform systemd that the mount is ready.

However, these are typically over-engineering for a standard bind mount. The Requires and After directives are designed precisely for managing these kinds of dependencies.

The Role of systemd-generator #

It’s also worth noting that systemd-generator can influence how .mount units are processed. These generators can create transient .mount units based on entries in /etc/fstab or other configurations. If you have an /etc/fstab entry for /volume1, ensure it’s correctly formatted and not causing delays.

A typical /etc/fstab entry for /volume1 might look like this:

/dev/md0 /volume1 ext4 defaults 0 0

Or, using UUID for greater reliability:

UUID=<your-md0-uuid> /volume1 ext4 defaults 0 0

If the nix.mount unit is being triggered by an fstab entry interpretation (which is not the case here, as you’re using a .mount file directly), then fstab processing order would matter. But since you have a dedicated .mount file, the direct After/Requires directives are the primary mechanism.

Troubleshooting the Device Interpretation #

The fact that systemctl status nix.mount shows What: /dev/md0 is the key indicator of the misinterpretation. This means that during the boot process, systemd has resolved /volume1/nix to /dev/md0 (or perhaps a path that ultimately points to /dev/md0’s filesystem mount point in a way that confused the bind mount logic).

This could happen if:

1. /volume1 is a mount point, and the bind mount is performed on a directory within that mount point. The system might be treating the bind mount as a re-mount of the underlying filesystem itself.
2. Timing: As discussed, the underlying filesystem for /volume1 might not be fully ready when the bind mount is attempted.

Considering /etc/fstab as a Fallback or Alternative #

Although you’ve opted for systemd .mount units, it’s worth noting how /etc/fstab handles bind mounts. A typical /etc/fstab entry for a bind mount looks like this:

/volume1/nix /nix none bind 0 0

systemd reads /etc/fstab and generates .mount units from it. If you were to use this /etc/fstab entry, systemd would create a transient run-volume1-nix.mount unit (or similar). The problem you’re facing is likely inherent to how systemd processes mount information, whether from .mount files or fstab. Thus, the dependency management (Requires/After) is crucial regardless of the configuration source.

Verifying Systemd Unit States and Dependencies #

To further diagnose, you can inspect the unit states and dependencies:

• Check volume1.mount status: systemctl status volume1.mount
• List all mount units: systemctl list-units --type=mount
• Analyze unit dependencies: systemctl list-dependencies --after nix.mount and systemctl list-dependencies --requires nix.mount

This will confirm if volume1.mount is active and if nix.mount is correctly ordered after it.

The Importance of noauto and nofail (and why they might not apply here) #

For devices in /etc/fstab that you don’t want mounted at boot, you would use noauto. For mounts that might fail but shouldn’t prevent boot, you’d use nofail. These options are not directly applicable to the Options= line in a .mount unit for a bind mount scenario, as you are actively wanting the mount to occur.

Finalizing the Correct .mount Unit Configuration #

Based on the analysis, the most robust solution for your scenario is to ensure the .mount unit correctly declares its dependencies on the underlying filesystem mount.

Recommended nix.mount Unit:

# /etc/systemd/system/nix.mount

[Unit]
Description=Nix Bind Mount to /volume1/nix
# Ensure the parent mount point /volume1 is active before attempting this bind mount.
# If /volume1 itself is managed by a systemd unit (e.g., /etc/systemd/system/volume1.mount),
# uncomment and adjust the following lines:
# Requires=volume1.mount
# After=volume1.mount
#
# If /volume1 is an fstab entry, systemd automatically creates a transient mount unit.
# The `After=local-fs.target` and `Requires=local-fs.target` ensure that all basic
# filesystems are available. If /volume1 is a network mount or has complex dependencies,
# you might need to be more specific with After= and Requires= directives.

[Mount]
What=/volume1/nix
Where=/nix
Type=none
Options=bind

[Install]
# This target ensures the mount is activated during the multi-user boot process,
# after local file systems are typically mounted.
WantedBy=local-fs.target

Key Considerations and Troubleshooting Steps:

1. Identify /volume1’s Mount Source: Determine precisely how /volume1 is mounted. Is it an /etc/fstab entry? Is it another .mount unit? Is it a network share?
2. Add Explicit Dependencies: If /volume1 is managed by a .mount unit (e.g., /etc/systemd/system/volume1.mount), you must include Requires=volume1.mount and After=volume1.mount in your nix.mount unit.
3. If /volume1 is an /etc/fstab Entry: systemd automatically processes /etc/fstab and creates transient mount units. The default ordering (local-fs.target) might not be sufficient if /volume1’s mount is delayed. In this case, you might need to add Requires=local-fs.target and After=local-fs.target to your nix.mount unit to ensure it runs after all general local file systems are available, which should include those from fstab.
4. Directory Existence: Ensure that both /volume1/nix and /nix directories exist before the unit is activated. The systemd mount process assumes the mount point (/nix) exists.
5. Filesystem Check: Run fsck on the filesystem that provides /volume1/nix to rule out any corruption that might cause delayed or inconsistent availability.
6. Log Analysis: After a reboot, carefully examine journalctl -xb for any error messages related to nix.mount or the mounting of /volume1.

The systemd-mount Example (Illustrative, not primary solution) #

While the .mount unit is preferred, if you were to use systemd-mount programmatically, the command might look like this (this is NOT a .mount unit, but how you’d execute it):

systemd-mount --bind /volume1/nix /nix

This command, when run manually, performs the bind mount correctly. The challenge is integrating this into the boot process reliably. A systemd service unit could wrap this command, but it adds complexity compared to a well-configured .mount unit.

Revisiting the What: /dev/md0 Observation #

The observation that systemctl status nix.mount shows What: /dev/md0 is the critical clue. It signifies that systemd or the underlying mount utility, when processing your unit, is resolving the source path to the block device instead of treating it as a directory to be bind-mounted. This often happens when the system is unsure how to interpret the path or when it detects that the path is already a mount point tied to a device.

By explicitly declaring dependencies like Requires=volume1.mount and After=volume1.mount, we are ensuring that the system knows /volume1 is a mount point that needs to be ready, and that /volume1/nix is a path within that mount. This context helps systemd correctly interpret the bind option.

A More Detailed .mount Unit with Explicit Device Path (Caution Advised)

In rare cases, if the dependency method doesn’t work, and you have absolute certainty about the underlying device for /volume1, you could theoretically try to inform systemd about it. However, this is generally discouraged as it makes the unit less portable and harder to manage if the device path changes.

# /etc/systemd/system/nix.mount (ADVANCED - USE WITH CAUTION)

[Unit]
Description=Nix Bind Mount to /volume1/nix (Device Explicit)
# This unit depends on the physical device containing /volume1 being mounted.
# Assuming /volume1 is mounted from /dev/md0 and has the correct filesystem type.
Requires=dev-md0.device   # Depends on the device node existing and being managed
After=dev-md0.device      # Starts after the device node is available

# Additionally, ensure the filesystem on /dev/md0 is mounted to /volume1
# If /volume1 is NOT managed by a systemd mount unit, systemd-mount from fstab may handle it.
# If /volume1 IS managed by a unit, add its dependencies here as well.
# For example:
# Requires=volume1.mount
# After=volume1.mount

[Mount]
What=/volume1/nix
Where=/nix
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

Why this is advanced and potentially problematic:

• dev-md0.device: This unit represents the device node itself. It doesn’t guarantee the filesystem on it is mounted.
• Device Path Stability: Relying on /dev/md0 is fragile. Device names can change. Using UUIDs is far more robust. However, systemd mount units for Type=none bind mounts typically expect directory paths, not device paths in What=.
• Complexity: This approach adds layers of dependency that might not be necessary if simpler dependency management (Requires/After on other .mount units) works.

The most elegant and robust solution remains ensuring the parent mount (/volume1) is correctly ordered and available.

Conclusion: Achieving Predictable Bind Mounts #

The discrepancy you’re encountering is a classic symptom of timing issues and how systemd resolves mount paths, particularly when dealing with bind mounts on existing mount points. By meticulously defining the dependencies of your nix.mount unit on the availability of its source directory’s parent mount point, you provide systemd with the necessary context to perform the bind mount accurately during boot.

The addition of Requires= and After= directives to link your nix.mount unit with the unit responsible for mounting /volume1 (or ensuring that /volume1 is available through standard fstab processing) is the most effective way to guarantee that /volume1/nix is correctly interpreted as the source for your bind mount, thus resolving the issue of it being incorrectly associated with /dev/md0. This approach ensures that your system is configured for stability and predictability, allowing you to confidently manage your filesystem hierarchy with bind mounts.

Remember to always perform a systemctl daemon-reload after modifying unit files and to test your configurations with a system reboot. This thorough approach, focusing on clear dependencies within the systemd ecosystem, is key to mastering complex system configurations and ensuring your services and mounts behave as expected, even in the intricate dance of the boot process.