Troubleshooting OS-Prober’s Incorrect UUID Generation for Arch Linux on a Multi-boot System

At revWhiteShadow, we understand the intricacies of multi-booting Linux distributions, especially when it comes to the seamless integration of boot menus. Recently, we encountered a perplexing issue where os-prober, a vital utility for detecting and configuring other operating systems within GRUB’s boot menu, began generating incorrect UUIDs specifically for our Arch Linux installation. This led to boot failures, prompting a deep dive into the underlying causes and effective solutions to restore os-prober’s accuracy for Arch Linux.

This article aims to provide a comprehensive guide, drawing from our extensive experience, to help you resolve os-prober issues with Arch Linux UUIDs, ensuring a stable and functional multi-boot environment. We will explore the potential reasons behind this anomaly and offer detailed steps to rectify the situation, allowing your boot manager to correctly identify and load your Arch Linux system.

Understanding the OS-Prober Mechanism and UUID Identification

OS-prober functions by scanning your system’s storage devices for other operating systems. It utilizes specific mechanisms to identify bootable partitions and their associated kernel parameters. A crucial aspect of this identification process is the use of Universally Unique Identifiers (UUIDs). UUIDs are unique alphanumeric strings assigned to file systems, providing a more robust and reliable way to identify partitions compared to device names like /dev/sda7, which can change depending on the boot order or hardware configuration.

When os-prober successfully identifies an Arch Linux installation, it typically generates a GRUB menu entry. This entry includes directives like linux /boot/vmlinuz-linux root=UUID=YOUR_CORRECT_UUID rw quiet and initrd /boot/initramfs-linux.img. The root=UUID=YOUR_CORRECT_UUID parameter is paramount; it tells the Arch Linux kernel where to find its root filesystem.

In the scenario we observed, the GRUB menu entry generated by os-prober for Arch Linux exhibited a critical discrepancy. While the initial search --fs-uuid --set=root commands correctly identified the Arch Linux partition’s UUID, the linux line itself contained a different and incorrect UUID for the root= parameter. This mismatch caused the boot process to fail, as the kernel was directed to an invalid location for its root filesystem.

The fact that os-prober continued to function correctly for other operating systems like openSUSE and Windows 7, and that the issue specifically impacted Arch Linux, pointed towards a configuration or environmental factor unique to the Arch installation or its interaction with os-prober. The addition of rw quiet to the boot line, which wasn’t present previously, further suggested a change in how os-prober was interpreting or constructing the Arch Linux boot entry.

Diagnosing the Root Cause: Why OS-Prober Might Report Wrong UUIDs for Arch Linux

Several factors can contribute to os-prober reporting incorrect UUIDs for a specific Linux distribution like Arch Linux. Understanding these potential causes is the first step towards a definitive solution.

1. Recent System Updates and Package Version Mismatches

The most common culprit for such anomalies, especially when a previously working system suddenly exhibits problems, is recent system updates. Arch Linux is a rolling-release distribution, meaning it receives frequent updates to its kernel, bootloader components, and system utilities.

Kernel Updates: A new kernel version might interact differently with os-prober or GRUB’s probing mechanisms. While GRUB is generally robust, subtle changes in how the kernel presents itself or how file systems are mounted during the probing phase could lead to misinterpretations by os-prober.
GRUB Updates: Updates to the GRUB bootloader itself, or its associated scripts and modules, could introduce regressions or new behaviors that affect how it interacts with os-prober.
OS-Prober Package Updates: If os-prober itself received an update, it might have introduced a bug or a change in its logic that specifically affects certain file system types or configurations prevalent in Arch Linux installations.
Systemd or related service changes: Arch Linux heavily relies on systemd. Changes in how systemd manages device nodes, mounts, or identifies file systems could potentially influence how os-prober gathers information.

It’s crucial to identify which packages were updated around the time the issue began. On Arch Linux, this information can often be found in /var/log/pacman.log. Reviewing this log can provide valuable clues about recent package installations, upgrades, or removals that might have triggered the problem.

2. File System Corruption or Inconsistencies

Although less likely if other OSes boot fine, subtle file system inconsistencies on the Arch Linux partition could theoretically confuse os-prober. While the system might still boot manually, the way os-prober reads metadata could be affected by minor corruption that isn’t immediately apparent during a standard boot.

Running fsck: Performing a file system check on the Arch Linux partition (from a live USB or another installed OS) is a good diagnostic step. However, ensure this is done when the partition is unmounted to prevent data loss.

3. GRUB Configuration Customizations and User Modifications

If you have made manual customizations to your GRUB configuration files beyond the standard output of grub-mkconfig, these changes could be interfering with os-prober’s operation.

GRUB_CUSTOM_LINUX and GRUB_CMDLINE_LINUX_DEFAULT: Modifications in /etc/default/grub or custom scripts in /etc/grub.d/ might be misinterpreted or conflict with the parameters os-prober is trying to set. The unexpected addition of rw quiet to the Arch Linux boot line strongly suggests that some part of the GRUB configuration generation process was altered.
Custom Scripts in /etc/grub.d/: These scripts are executed by grub-mkconfig to generate the final grub.cfg. If a custom script intended for Arch Linux is incorrectly written or designed to override certain parameters, it could lead to the observed behavior.

4. Interaction with Other Boot Loaders or Partitioning Schemes

While you mentioned Ubuntu manages GRUB, the presence of multiple operating systems, different partition tables (like MBR or GPT), and potentially other boot managers (even if dormant) can sometimes create complex interactions.

UEFI vs. BIOS: The boot mode (UEFI or Legacy BIOS) can influence how boot information is stored and accessed. Ensure GRUB and your OS installations are consistent in their boot mode if possible, or at least understand how they are interacting.
Partition Table Type: While generally handled well, complex partition layouts or unusual configurations might expose edge cases in os-prober.

5. Specific Arch Linux Kernel/Initramfs Configuration

Arch Linux allows for a high degree of customization. The way its kernel and initramfs are configured, particularly regarding modules loaded or parameters passed during early boot, could be a factor.

mkinitcpio.conf: Changes to /etc/mkinitcpio.conf and the subsequent rebuilding of the initramfs (using mkinitcpio -P) might influence how the system identifies its root device during the early boot stages, which os-prober might indirectly be affected by.

Step-by-Step Solutions to Fix OS-Prober’s Incorrect UUID for Arch Linux

Based on our analysis, we can implement a series of targeted solutions to rectify the os-prober incorrect UUID for Arch Linux issue.

Step 1: Verify and Correct the Arch Linux UUID Manually (Temporary Fix)

Before attempting to fix os-prober’s behavior, it’s essential to confirm the correct UUID for your Arch Linux root partition.

Boot into Arch Linux using the manual method you discovered (e.g., specifying /dev/sda7).
Once booted, open a terminal and run:
```
lsblk -f
```
or
```
blkid
```
Locate your Arch Linux root partition (e.g., /dev/sda7) and note its correct UUID.
Now, manually edit your GRUB configuration file. This is typically /boot/grub/grub.cfg. However, direct editing of grub.cfg is strongly discouraged as it will be overwritten by grub-mkconfig. The correct approach is to edit /etc/default/grub and then regenerate the GRUB configuration.

Important Note: While you can temporarily fix the boot by manually editing /boot/grub/grub.cfg, the goal is to fix the underlying issue that causes os-prober to generate it incorrectly.

Step 2: Reconfigure GRUB and Regenerate the Configuration

The most reliable way to ensure your GRUB configuration is up-to-date and correctly reflects your system is to regenerate it. This process utilizes os-prober to detect other operating systems.

Ensure os-prober is enabled:
- Open the GRUB configuration file:
```
sudo nano /etc/default/grub
```
- Make sure the following line is present and uncommented:
```
GRUB_DISABLE_OS_PROBER=false
```
  If it’s set to true, change it to false.
Generate the new GRUB configuration:
- Run the command:
```
sudo grub-mkconfig -o /boot/grub/grub.cfg
```
- This command scans your system for other OSes using os-prober and reconstructs the /boot/grub/grub.cfg file.
Reboot your system and check if the Arch Linux entry in the GRUB menu now displays the correct UUID.

Step 3: Investigate and Correct Custom GRUB Scripts

If regenerating the GRUB configuration doesn’t resolve the issue, custom scripts in /etc/grub.d/ might be interfering.

Review custom scripts:
- Navigate to /etc/grub.d/.
- Look for any files that seem to be related to Arch Linux or boot entry generation. Files are executed in numerical order. Common custom scripts might be named like 10_custom, 30_os-prober, or custom numbered files.
- Pay close attention to any scripts that might be overriding default parameters or adding specific options like rw quiet in an unintended way.
Temporarily disable custom scripts:
- If you suspect a custom script is the cause, you can temporarily disable it by changing its permissions to make it non-executable:
```
sudo chmod -x /etc/grub.d/your_custom_script_name
```
- After disabling, regenerate the GRUB configuration again:
```
sudo grub-mkconfig -o /boot/grub/grub.cfg
```
- Test if the Arch Linux entry is now correct. If it is, you’ve found the offending script and can proceed to debug or rewrite it.

Ensure that all components related to GRUB and system detection are up-to-date.

Update your Arch Linux system:
```
sudo pacman -Syu
```
This will update the kernel, GRUB itself, and any other system packages. This is often the most effective solution if a package update caused the problem.
Reinstall GRUB (if necessary): If you suspect GRUB installation itself might be corrupted, you can reinstall it. First, identify your EFI System Partition (ESP) if using UEFI, or your boot drive if using BIOS.
- For UEFI systems: Assuming your ESP is mounted at /boot/efi:
```
sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB
```
  (Adjust --bootloader-id if you use a different identifier).
- For BIOS systems: Assuming your boot drive is /dev/sda:
```
sudo grub-install /dev/sda
```
- After reinstalling GRUB, always regenerate the configuration:
```
sudo grub-mkconfig -o /boot/grub/grub.cfg
```

Step 5: Examine OS-Prober Configuration and Dependencies

os-prober itself has configuration options and relies on certain system utilities to function correctly.

Check os-prober’s configuration:
- The primary configuration for os-prober within the GRUB environment is often handled through scripts in /etc/grub.d/. There isn’t a separate os-prober.conf file in the same way as other systems.
- However, ensure that necessary packages for file system detection are installed. For example, if your Arch Linux partition is ext4, ensure the e2fsprogs package is installed. For btrfs, ensure btrfs-progs is installed. pacman -Syu usually handles this, but it’s good to be aware.
Look for os-prober logs or debug output: While GRUB doesn’t typically provide detailed os-prober logs in grub.cfg generation, you might find more information if you try to run os-prober manually in a debug mode, although this is more advanced and might not be straightforward.

Step 6: Address the `rw quiet` Parameter Anomaly

The addition of rw quiet to the Arch Linux boot line is a specific symptom that needs attention. This parameter is typically part of the GRUB_CMDLINE_LINUX_DEFAULT setting in /etc/default/grub.

Review /etc/default/grub:
- Open the file:
```
sudo nano /etc/default/grub
```
- Look for the line starting with GRUB_CMDLINE_LINUX_DEFAULT.
- If this line already contains rw quiet and you didn’t add it, it might be coming from a custom script in /etc/grub.d/ that is being processed before or in a way that overrides the default settings.
- If the line is clean (e.g., GRUB_CMDLINE_LINUX_DEFAULT="quiet"), then a custom script is almost certainly the culprit for adding rw quiet.
If a custom script is adding rw quiet:
- As mentioned in Step 3, identify the script and either remove the line responsible for adding rw quiet or modify it.
If /etc/default/grub is the source:
- Ensure your intended parameters are correctly listed. If you want Arch to boot with rw quiet, it should be there. If you don’t want it, remove it from /etc/default/grub.
- Crucially, after any change to /etc/default/grub or /etc/grub.d/ files, you must regenerate the GRUB configuration:
```
sudo grub-mkconfig -o /boot/grub/grub.cfg
```

Step 7: Recreate Arch Linux Initramfs

In some rare cases, the initramfs might be generated with parameters that somehow influence how os-prober or GRUB perceives the root filesystem. Recreating it can resolve such subtle issues.

Regenerate the initramfs for Arch Linux:
```
sudo mkinitcpio -P
```
This command rebuilds all initramfs images defined in /etc/mkinitcpio.conf.
After regenerating initramfs, regenerate GRUB configuration:
```
sudo grub-mkconfig -o /boot/grub/grub.cfg
```
Reboot and test.

Advanced Troubleshooting: Deep Dive into OS-Prober’s Behavior

If the above steps haven’t fully resolved the os-prober incorrect UUID for Arch Linux problem, we need to delve deeper into how os-prober functions and how GRUB uses its output.

Understanding `os-prober` Output and GRUB’s Interpretation

When grub-mkconfig runs, it invokes os-prober for each detected operating system. os-prober then outputs lines that GRUB interprets to build its menu entries. For Linux systems, a typical output from os-prober might look something like this (simplified):

menuentry 'Arch Linux' --class arch --class gnu-linux --class os {
    insmod part_gpt
    insmod ext2
    search --no-floppy --fs-uuid --set=root YOUR_CORRECT_ARCH_UUID
    # ... other search hints if applicable
    linux /boot/vmlinuz-linux root=UUID=YOUR_CORRECT_ARCH_UUID rw quiet
    initrd /boot/initramfs-linux.img
}

The discrepancy you observed suggests that either: a) os-prober is reporting the wrong UUID to grub-mkconfig for the linux line’s root= parameter. b) grub-mkconfig or a script within /etc/grub.d/ is modifying the output from os-prober before writing it to grub.cfg.

The fact that the search --fs-uuid --set=root command uses the correct UUID, but the linux /boot/vmlinuz-linux root=UUID=... line uses the incorrect one, strongly points towards the second scenario. A script is likely injecting or overriding the root= parameter.

Debugging Custom Scripts in `/etc/grub.d/`

This is where we focus if Step 3 indicated a custom script might be the cause.

Identify the relevant script: As mentioned, files in /etc/grub.d/ are executed in order. The 30_os-prober script is the standard one that handles other OSes. If you have custom scripts numbered before or around 30_os-prober (e.g., 10_arch_custom, 20_custom), they might be influencing the output.
Temporarily comment out suspicious lines:
- Edit the suspected custom script (e.g., sudo nano /etc/grub.d/10_custom).
- Find lines that might be adding or modifying kernel parameters. These often look like:
```
menuentry "..." {
    ...
    # Example of a problematic line:
    # echo 'linux /boot/vmlinuz-linux root=UUID=AN_INCORRECT_UUID rw quiet'
    # Or potentially modifying variables used by os-prober scripts
}
```
- Comment out these lines using #.
- Save the file and regenerate GRUB: sudo grub-mkconfig -o /boot/grub/grub.cfg.
- Test. Repeat this process for different custom scripts if necessary.

Understanding the `rw quiet` Addition

The rw quiet parameters are standard kernel boot parameters. rw means the root filesystem is mounted read-write, and quiet suppresses most kernel messages during boot.

If these were not present before and suddenly appeared, it strongly suggests a change in the default configuration that generates these parameters. This default is usually handled within the 30_os-prober script itself or via system-wide GRUB configuration.
If you inspect the 30_os-prober script (usually located in /etc/grub.d/30_os-prober), you might find logic that adds rw quiet by default for detected Linux systems. It’s possible that a recent update to os-prober or GRUB’s scripts changed this default behavior.

Final Verification and System Health Checks

After implementing solutions, it’s crucial to perform final checks.

Verify UUIDs in grub.cfg: Boot into Ubuntu (or your GRUB managing OS), navigate to /boot/grub/grub.cfg, and open the Arch Linux menu entry. Double-check that the root=UUID= parameter in the linux line matches the actual UUID of your Arch partition obtained via lsblk -f or blkid while booted into Arch.
Test booting into Arch Linux: Reboot your system and select the Arch Linux entry from the GRUB menu. Confirm that it boots successfully without any errors related to the root filesystem.
Test booting into other OSes: Ensure that the changes made did not negatively impact the boot entries for Ubuntu, openSUSE, or Windows 7.

By systematically addressing potential causes, from simple updates to complex script interactions, we can effectively resolve the os-prober incorrect UUID issue for Arch Linux, ensuring a stable and reliable multi-boot environment. Our approach at revWhiteShadow prioritizes understanding the underlying mechanisms and applying precise solutions to achieve optimal system performance.

os-prober giving wrong UUID for Arch Linux

Troubleshooting OS-Prober’s Incorrect UUID Generation for Arch Linux on a Multi-boot System #

Understanding the OS-Prober Mechanism and UUID Identification #

Diagnosing the Root Cause: Why OS-Prober Might Report Wrong UUIDs for Arch Linux #

1. Recent System Updates and Package Version Mismatches #

2. File System Corruption or Inconsistencies #

3. GRUB Configuration Customizations and User Modifications #

4. Interaction with Other Boot Loaders or Partitioning Schemes #

5. Specific Arch Linux Kernel/Initramfs Configuration #

Step-by-Step Solutions to Fix OS-Prober’s Incorrect UUID for Arch Linux #

Step 1: Verify and Correct the Arch Linux UUID Manually (Temporary Fix) #

Step 2: Reconfigure GRUB and Regenerate the Configuration #

Step 3: Investigate and Correct Custom GRUB Scripts #

Step 4: Update GRUB and Related Packages #

Step 5: Examine OS-Prober Configuration and Dependencies #

Step 6: Address the rw quiet Parameter Anomaly #

Step 7: Recreate Arch Linux Initramfs #

Advanced Troubleshooting: Deep Dive into OS-Prober’s Behavior #

Understanding os-prober Output and GRUB’s Interpretation #

Debugging Custom Scripts in /etc/grub.d/ #

Understanding the rw quiet Addition #

Final Verification and System Health Checks #