mkdir Says Folder Exists Even Though It Doesn’t Show with ls -a: Troubleshooting CIFS Mount Issues

At revWhiteShadow, we, revWhiteShadow and kts, are dedicated to providing insightful solutions to common Linux and system administration challenges. Today, we tackle a perplexing issue encountered when working with CIFS mounts on Arch Linux: the mkdir command reporting that a directory exists when it seemingly doesn’t, as verified by ls -a. This can be particularly frustrating when attempting to create new directories within a mounted network share, and standard troubleshooting steps don’t reveal the root cause. This guide dives deep into potential causes and offers effective solutions.

Understanding the Problem: CIFS Mounts and “File Exists” Errors

The core of the issue lies in a discrepancy between the client’s (your Arch Linux system) view of the file system and the server’s (your NAS device) actual state. This inconsistency can be triggered by various factors specific to CIFS (Common Internet File System) mounts and their interactions with the underlying network and file system layers.

Why ls -a Might Deceive You

The ls -a command lists all files and directories, including hidden ones (those starting with a dot .). However, it relies on the information provided by the CIFS mount, which, in turn, gets its data from the NAS server. If there’s a problem in this communication chain, the ls -a output might not accurately reflect the server’s file system.

Common Causes and Solutions

We will now explore the most likely causes and corresponding solutions in a detailed way, from the basics to the most unusual scenarios.

1. The Dreaded Filename Case Sensitivity Issue

CIFS, by default, can be case-insensitive. This means that “Subdir” and “subdir” might be treated as the same entity. If, for example, a file named “subdir” already exists (perhaps created through a different client or interface on your NAS), attempting to create a directory named “subdir” will result in the “File exists” error, even if the shell visually represents them as different.

Solution:

  1. Verify Existing Files: Double-check for files or directories with names that differ only in case within the target directory on the NAS. Use the NAS’s web interface or SSH into the NAS to ensure that the files and directories are properly displayed.

  2. Consistent Naming: Adopt a consistent naming convention to avoid accidental case collisions.

  3. Mount Options (Server-Side): In some cases, you can influence case sensitivity on the server side, although this depends on the NAS’s capabilities and configuration. Consult your NAS documentation.

2. The CIFS Cache Staleness Problem

CIFS clients maintain a cache of file and directory information to improve performance. This cache can become stale, meaning it contains outdated information about the server’s file system. In our scenario, the cache might be missing the existence of the new directory or incorrectly flagging a name as already taken.

Solution:

  1. Unmount and Remount: The simplest and often most effective solution is to unmount and remount the CIFS share:

    sudo umount /path/to/local/dir
sudo mount /path/to/local/dir

    This forces the client to refresh its cache with the latest information from the server.

  2. cache=none Mount Option (Use with Caution): As a more aggressive approach, you can try mounting the share with the cache=none option. This disables client-side caching. This is useful for debugging.

    sudo mount -t cifs //IP_ADDRESS/path/to/dir /path/to/local/dir -o uid=my_user,gid=my_group,dir_mode=0777,file_mode=0777,credentials=path/to/my/creds,cache=none

    Important: Disabling caching can significantly degrade performance, so only use this for troubleshooting. If it resolves the issue, investigate further and look for a less impactful solution.

3. Intermittent Network Issues and Packet Loss

CIFS relies on a stable network connection. Intermittent network glitches or packet loss during file system operations can lead to incomplete or corrupted metadata updates. In our case, the server might have partially created the directory, but the client didn’t receive confirmation, leading to the “File exists” error on subsequent attempts.

Solution:

  1. Test Network Connectivity: Use ping or traceroute to check for packet loss or high latency between your Arch Linux system and the NAS.

    ping IP_ADDRESS
traceroute IP_ADDRESS

  2. Check Network Cables and Hardware: Ensure your network cables are properly connected and functioning. If using Wi-Fi, consider switching to a wired connection for more stability.

  3. Firewall Rules: Verify that no firewall rules on your Arch Linux system or the NAS are blocking CIFS traffic. The necessary ports for CIFS (typically ports 137, 138, 139, and 445) must be open.

  4. MTU Size: In some rare cases, the Maximum Transmission Unit (MTU) size on your network interface might be too large, leading to fragmentation and packet loss. Try reducing the MTU size on your Arch Linux system.

    sudo ip link set eth0 mtu 1400 # Replace eth0 with your network interface name

    Test different MTU values (e.g., 1400, 1450, 1492) to find the optimal setting.

4. Inconsistent Permissions on the NAS

While your fstab entry specifies dir_mode=0777 and file_mode=0777, the underlying permissions on the NAS itself might be more restrictive. The effective permissions are the intersection of the CIFS mount permissions and the NAS’s file system permissions.

Solution:

  1. NAS Permissions Check: Access the NAS’s web interface or use SSH to examine the permissions of the parent directory /path/to/dir. Ensure that the user associated with my_user (as specified in your fstab) has write access to the parent directory.
  2. ACLs (Access Control Lists): If your NAS supports ACLs, they might be overriding the basic file permissions. Investigate the ACL settings for the parent directory and ensure that my_user has the necessary permissions.
  3. User Mapping: Double-check that the user mapping between your Arch Linux system and the NAS is correctly configured. The uid and gid options in your fstab should correspond to the user and group IDs on the NAS.

5. The “Tombstone” Issue (Advanced)

In certain situations, particularly with older CIFS implementations or specific NAS configurations, a “tombstone” entry might be created for a deleted file or directory. This tombstone entry prevents the creation of a new file or directory with the same name, even though it’s not visible through standard file listings.

Solution:

This is a tricky situation that often requires server-side intervention.

  1. NAS Reboot: A reboot of the NAS can sometimes clear tombstone entries. This is a simple first step to try.
  2. Server-Side Tools (If Available): Some NAS devices provide command-line tools or web interface options for managing CIFS shares and clearing stale metadata. Consult your NAS documentation for details.
  3. Samba Configuration (Advanced): If your NAS uses Samba as its CIFS server, you might be able to adjust Samba’s configuration to control tombstone behavior. This requires advanced knowledge of Samba configuration.

6. Open Files and Locks

Another less frequent cause is that some other client might have an open handle to the directory or a file within it, thereby creating a lock.

Solution:

  1. Close Open Files: Check if other users or processes have files open inside the mount point that may be preventing creation.
  2. Restart Samba/NAS Processes: As a last resort, restarting the Samba services on the NAS or rebooting it clears all locks and ensures all processes start fresh.

Debugging with mount.cifs Options

The mount.cifs command provides various debugging options that can help pinpoint the root cause of the issue.

vers=x.y:

Specifies the SMB protocol version. Try different versions (e.g., vers=3.0, vers=2.1, vers=1.0) to see if one works more reliably with your NAS. The latest versions generally provide the best performance and security, but older versions might be necessary for compatibility with older NAS devices.

sec=mode:

Specifies the security mode. Try different modes (e.g., sec=ntlmssp, sec=ntlmi, sec=ntlm) if you’re having authentication issues.

debug:

Enables verbose debugging output. This can generate a lot of data, but it can be helpful for diagnosing complex issues. Capture the output to a file for analysis.

sudo mount -t cifs //IP_ADDRESS/path/to/dir /path/to/local/dir -o uid=my_user,gid=my_group,dir_mode=0777,file_mode=0777,credentials=path/to/my/creds,debug > mount.log 2>&1

Analyze the mount.log file for error messages or clues about the problem.

Using tcpdump or Wireshark

For even deeper debugging, you can use network packet capture tools like tcpdump or Wireshark to analyze the traffic between your Arch Linux system and the NAS. This allows you to inspect the SMB protocol messages and identify any errors or inconsistencies.

Addressing the Specific Case: /mnt/nas/SteamLibrary/steamapps/downloading

Given the specific commands and output provided, let’s analyze the situation.

  • ls -la /mnt/nas/SteamLibrary/steamapps/downloading returns “No such file or directory”, indicating that the directory is genuinely missing from the client’s perspective.
  • mkdir /mnt/nas/SteamLibrary/steamapps/downloading returns “File exists”, suggesting a conflict.

Here’s a targeted approach:

  1. Server-Side Confirmation: Log in to your NAS and absolutely confirm that the downloading directory doesn’t exist within /path/to/SteamLibrary/steamapps (replace with your actual NAS path). Use the NAS’s file manager or SSH to verify. Do not rely on the client-side view.
  2. Tombstone Suspicion: Given that ls shows no directory but mkdir claims it exists, a tombstone entry is highly probable. Try restarting the NAS.
  3. Cache Clearing (Again): After the NAS reboot, unmount and remount the CIFS share on the client side to ensure a completely fresh cache.
  4. Permissions Audit: Verify that the jsmith user (on the NAS) has full read/write/execute permissions for /path/to/SteamLibrary/steamapps (replace with your actual NAS path). ACLs could be interfering.

Conclusion

The “mkdir: cannot create directory: File exists” error on CIFS mounts, despite ls -a showing otherwise, is a common yet frustrating problem. By systematically working through the causes above from case sensitivity to tombstone entries and employing debugging tools, you can diagnose and resolve the issue, regaining access to your network shares. At revWhiteShadow, we strive to provide you with the insights necessary to tackle any system administration challenge.