Why Container Processes on Debian Trixie Aren’t Visible from the Host: A Deep Dive

As revWhiteShadow, a kts personal blog site, we’ve encountered a perplexing situation while migrating our containerized Oracle databases from Debian 12 (Bookworm) to Debian 13 (Trixie). On Debian 12, using the ps command on the host system, we could easily view the processes running within the container. However, after upgrading to Debian 13, this was no longer the case. This article explores the reasons behind this change in behavior and provides solutions for restoring the visibility of container processes from the host.

Understanding Process Isolation in Containers

Containers, fundamentally, leverage kernel features like namespaces and cgroups to isolate processes. Namespaces provide a level of abstraction, creating isolated environments for process IDs (PIDs), network interfaces, mount points, inter-process communication (IPC), and user IDs (UIDs). This isolation prevents processes within a container from directly accessing or interfering with processes running on the host or in other containers.

Cgroups (control groups), on the other hand, limit the resources a container can consume, such as CPU, memory, disk I/O, and network bandwidth. While cgroups primarily focus on resource management, they also play a role in process isolation by restricting a container’s access to system resources.

The combination of namespaces and cgroups creates a secure and isolated environment for containerized applications, ensuring that they don’t negatively impact the host system or other containers.

The Observed Discrepancy: Debian 12 vs. Debian 13

The core issue is the difference in process visibility between Debian 12 and Debian 13. Let’s reiterate the scenario:

  • Inside the container: Using ps -ef | grep -i ora, we can see the Oracle processes running correctly, including tnslsnr, db_pmon_FREE, and other database-related processes.
  • On the Debian 13 host: Running the same command, ps -ef | grep -i ora, only reveals the tnslsnr process, but not the other Oracle database processes.
  • On the old Debian 12 host: Running the same command ps -ef | grep -i ora, we can see all the Oracle database related processes.
  • Mount Information: mount | grep -i proc confirms that the /proc filesystem is mounted on both the host and within the container, indicating that this isn’t a standard procfs mounting issue.
  • Kernel Command Line: The kernel command line (obtained from /proc/cmdline) doesn’t show any specific configurations that would intentionally hide container processes.

This behavior suggests a change in the default containerization settings or kernel behavior between Debian 12 and Debian 13 that affects how processes in a PID namespace are exposed to the host system.

Investigating Potential Causes and Solutions

Several factors could contribute to this difference in process visibility. We’ll explore each possibility in detail:

1. Kernel Version Differences and PID Namespace Handling

The kernel version plays a crucial role in how PID namespaces are implemented and managed. Debian 13 ships with a newer kernel version than Debian 12. Subtle changes in the kernel’s PID namespace handling might explain why processes are less visible. It is important to investigate kernel release notes for changes to namespace behavior.

Solution: While downgrading the kernel isn’t practical, examining the kernel configuration options related to PID namespaces could offer clues. Configuration options related to CONFIG_PID_NS are of importance. Unfortunately, there is not much can be done on this side, since Debian 13 ships with an stable Kernel release.

2. Container Runtime Configuration (Docker, containerd, etc.)

The container runtime used (e.g., Docker, containerd, CRI-O) configures the container’s environment, including the PID namespace. It is likely that the default settings for PID namespace isolation have become more restrictive in newer versions of these runtimes.

Investigating Docker:

If using Docker, inspect the Docker daemon configuration file (/etc/docker/daemon.json). Look for parameters related to userns-remap or other security-related settings that might affect process visibility. Also, check the Docker container’s configuration when creating it. Parameters like --pid=host can drastically alter PID namespace behavior, but this setting is likely not the default and would be unusual for database containers.

Solution (Docker):

  • Disable User Namespace Remapping: If userns-remap is enabled, temporarily disable it to see if it resolves the issue. However, be aware that disabling user namespace remapping can introduce security risks.

  • Use nsenter for Direct Inspection: The nsenter command allows you to enter the container’s PID namespace directly, bypassing any isolation issues. To use it, you’ll need the container’s PID:

    docker inspect -f '{{.State.Pid}}' <container_name_or_id>
nsenter -n -p -t <container_pid> ps -ef | grep ora

    If nsenter shows the Oracle processes, but ps on the host doesn’t, it reinforces the idea that the host’s process view is being filtered.

Investigating containerd:

If using containerd directly (without Docker), examine the containerd configuration file (typically /etc/containerd/config.toml). Pay attention to the runtimes section and any settings related to the OCI runtime used (usually runc). The OCI runtime configuration might be influencing PID namespace isolation.

Solution (containerd):

  • OCI Runtime Configuration: The OCI runtime (runc) has its own configuration options. You’ll need to consult the runc documentation to understand how to modify its behavior regarding PID namespaces. Usually, the containerd’s configuration file points to the directory where to put OCI specifications.

3. Systemd and Process Cgroups

Systemd, the system and service manager, plays a significant role in process management on modern Linux systems. It uses cgroups to manage processes and can influence process visibility, especially within containers.

Investigating Systemd:

Check the container’s systemd configuration. While less common for database containers, ensure that systemd isn’t configured in a way that actively hides processes from the host. Look for settings like PrivateTmp, PrivateDevices, or ProtectSystem in the service files (e.g., /etc/systemd/system/your_container_service.service) that might indirectly affect process visibility.

Solution (Systemd):

  • Review Service Files: If you’re using systemd to manage the container, carefully review the service files for any settings that could limit process visibility. Experiment by commenting out or modifying potentially restrictive settings and restarting the container to see if it resolves the issue.
  • Systemd-nspawn considerations: If you are using systemd-nspawn to run the container, you need to investigate the --private-users flag that hides container users from host.

4. Procfs Mount Options and Kernel Hardening

While you’ve confirmed that /proc is mounted, it’s worth revisiting the mount options used. Specific mount options, especially those related to security hardening, could influence process visibility. Some kernel hardening techniques involve restricting access to the /proc filesystem to prevent information leakage.

Investigating Procfs Mount Options:

Double-check the output of mount | grep proc on both the host and within the container. Look for options like hidepid= or gid=. The hidepid= option, in particular, controls who can see process information in /proc.

  • hidepid=1: Prevents processes from seeing processes owned by other users.
  • hidepid=2: Prevents processes from seeing any other processes in /proc, regardless of the user ID.
  • gid=: Specifies a group ID that is allowed to see all processes.

Solution (Procfs):

If hidepid=2 is set, this is almost certainly the cause of the problem. You’ll need to modify the mount options for /proc. However, directly remounting /proc on the host is generally not recommended and can lead to system instability.

A safer approach is to adjust the container runtime’s configuration to control how /proc is mounted within the container. For example, in Docker, you could use a bind mount to override the default /proc mount:

docker run -d --name my_container -v /proc:/proc:ro <image_name>

Warning: Modifying /proc mount options can have significant security implications. Understand the risks before making changes.

5. Security Modules (AppArmor, SELinux)

Security modules like AppArmor and SELinux enforce mandatory access control policies, which can restrict a process’s ability to access resources, including information about other processes in /proc.

Investigating Security Modules:

Check the status of AppArmor and SELinux on both the host and within the container.

  • AppArmor: Use apparmor_status to check the status of AppArmor profiles. Look for profiles that might be restricting access to /proc.
  • SELinux: Use getenforce to check if SELinux is enabled and in enforcing mode. If it is, use ausearch -m avc to search for SELinux denials related to ps or /proc.

Solution (Security Modules):

If a security module is interfering, you’ll need to modify its configuration to allow ps to access the necessary information in /proc. This typically involves creating or modifying AppArmor profiles or SELinux policies.

  • AppArmor: Create a custom AppArmor profile for ps that allows it to read the contents of /proc/[pid]. The exact syntax of the profile depends on the version of AppArmor.
  • SELinux: Create a custom SELinux policy module that allows ps to access the proc_t type. You’ll need to use the audit2allow tool to generate the policy module based on the SELinux denials.

Important: Modifying security module configurations requires a thorough understanding of their policies and can weaken system security if done incorrectly.

6. The pid_max Kernel Parameter

The /proc/sys/kernel/pid_max file determines the maximum PID value. If a container process has a PID higher than pid_max on the host, it might not be visible.

Investigating pid_max:

Check the value of /proc/sys/kernel/pid_max on both the host and within the container.

Solution (pid_max):

If the pid_max value on the host is significantly lower than the PID of the container processes, you can increase it by writing a new value to the file:

echo 4194304 > /proc/sys/kernel/pid_max

However, this is unlikely to be the root cause, as modern systems typically have a reasonably large pid_max value.

7. PR_SET_PID_NS

The PR_SET_PID_NS operation, when applied through prctl(), allows a process to move itself to a new PID namespace. Although not usually something applied naively, it’s worth keeping in mind.

Solution (PR_SET_PID_NS): If an unusual case occurs and a process moves itself to another PID namespace, then this function must be audited.

8. Utilizing cgroupfs to monitor processes

If the processes are still invisible via the methods mentioned above, it is important to note that cgroupfs can still be leveraged to filter processes.

Solution (cgroupfs):

  1. Mount the cgroup filesystem if it’s not already mounted:

    mount -t cgroup2 cgroup2 /sys/fs/cgroup

  2. Navigate to the cgroup directory for the container in question. The naming convention depends on the container runtime. For Docker, it is usually /sys/fs/cgroup/docker/<container_id>.

  3. List the processes associated with the cgroup:

    cat /sys/fs/cgroup/docker/<container_id>/cgroup.procs

This gives you direct access to the PIDs registered under the cgroup, irrespective of what ps may show due to namespace isolation.

Conclusion: A Multi-Faceted Problem

The lack of visibility of container processes on Debian 13 compared to Debian 12 is likely due to a combination of factors, with the most probable culprits being changes in the default configuration of container runtimes (Docker, containerd) and the kernel’s handling of PID namespaces. Security modules like AppArmor and SELinux could also play a role.

We recommend systematically investigating each of the potential causes outlined above, starting with the container runtime configuration and then moving on to the kernel, systemd, and security modules. By carefully examining and adjusting these settings, you should be able to restore the desired visibility of container processes on your Debian 13 host.

Remember that any changes to security-related settings should be made with caution and a thorough understanding of the potential risks. Always test changes in a non-production environment before applying them to production systems.

We hope this comprehensive guide helps you troubleshoot and resolve the issue of container process visibility on Debian Trixie. Good luck!