Debian Trixie: Troubleshooting LXC Unprivileged Container Issues

Debian Trixie, the latest stable release, introduces numerous improvements and updated packages. However, migrating to a new system can sometimes present unexpected challenges. As revWhiteShadow, along with kts personal blog site, we’ve encountered and addressed various issues related to unprivileged LXC containers on Debian Trixie, and we’re sharing our findings to help you navigate these potential pitfalls. Like many users, we found that upgrading to Debian Trixie from Bullseye, where LXC worked flawlessly, resulted in broken LXC unprivileged containers.

Symptoms of LXC Issues on Debian Trixie

The primary symptom we observed was the inability to properly manage and interact with unprivileged LXC containers. This manifested in several ways:

lxc-ls Errors: The lxc-ls command, used to list containers, produced errors related to namespace switching, specifically:
```
lxc-ls: ../src/lxc/utils.c: switch_to_ns: 900 Operation not permitted - Failed to set process <PID> to "net" of 3
```
This error indicates a problem with the process’s ability to switch to the network namespace associated with the container. These kind of errors indicates a problem with privileges.

lxc-attach Failures: Attempting to attach to a running container using lxc-attach resulted in errors like:

lxc-attach: dns1: ../src/lxc/cgroups/cgfsng.c: enter_scope: 1335 Failed opening dbus connection
lxc-attach: dns1: ../src/lxc/cgroups/cgfsng.c: cgroup_attach_move_into_leaf: 2859 Failed entering scope 'lxc-dns1-0.scope'
lxc-attach: dns1: ../src/lxc/cgroups/cgfsng.c: cgroup_attach_move_into_leaf: 2897 Permission denied - Failed to move process into target cgroup via fd 7 and 8
lxc-attach: dns1: ../src/lxc/conf.c: userns_exec_minimal: 4685 Permission denied - Running parent function failed
lxc-attach: dns1: ../src/lxc/attach.c: do_attach: 1238 No data available - Failed to receive lsm label fd
lxc-attach: dns1: ../src/lxc/attach.c: do_attach: 1376 Failed to attach to container

These errors point to issues with cgroup configuration, particularly around permission and access to the container’s cgroup scope. It also seems that the LSM label is not correctly provided.

Container Shutdowns: After some time, containers would unexpectedly stop, transitioning to a STOPPED state.
Failure to mount API Filesystems: When debugging using lxc-start -n dns1 -F -l DEBUG -o /tmp/lxc-nomecontainer.log produced the following error message:
```
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
```
This indicates a major problem with the cgroup setup and permissions.

Root Cause Analysis: Understanding the Problem

The core issue stems from changes in how Debian Trixie handles cgroups, specifically with unprivileged containers. While lxc-checkconfig may report a seemingly functional setup, deeper inspection reveals permission problems preventing proper container operation.

The switch to cgroupfs unified hierarchy, or cgroup v2, which is now the default in Trixie, can cause issues with older LXC configurations designed for cgroup v1. This is because cgroup v2 enforces stricter security and resource management policies, which can clash with the way unprivileged containers traditionally operate.

The error messages indicate that the unprivileged user attempting to manage the containers lacks the necessary permissions to perform actions within the cgroup hierarchy, such as:

Creating or joining cgroup scopes
Moving processes into cgroups
Accessing necessary systemd API filesystems within the container environment

Furthermore, AppArmor or SELinux profiles (if enabled) might be interfering with the container’s ability to perform certain operations, especially those related to namespace switching and cgroup management.

Solutions and Workarounds: Getting LXC Working on Debian Trixie

Several approaches can be taken to resolve these issues. We recommend trying these in order, as some solutions may be more appropriate or effective depending on your specific configuration and security requirements.

1. Disabling Systemd Unified Cgroup Hierarchy (Not Recommended)

A temporary workaround, but strongly discouraged for production environments, is to disable the systemd unified cgroup hierarchy. This reverts to the older cgroup v1 system. To do this:

Edit /etc/default/grub:
```
sudo nano /etc/default/grub
```
Add the following kernel parameter to the GRUB_CMDLINE_LINUX line:
```
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0"
```
Update GRUB and reboot:
```
sudo update-grub
sudo reboot
```

Warning: Disabling cgroup v2 significantly reduces system security and resource management capabilities. This is not a long-term solution and should only be used for testing or as a temporary measure until a proper fix is implemented. In particular, running cgroup v1 on Debian Trixie may introduce instabilities. If you choose to run with cgroup v1, you are on your own.

2. Adjusting LXC Configuration for Cgroup v2

The preferred solution is to adapt your LXC container configurations to be compatible with cgroup v2. This involves several steps:

a. Ensuring LXC Version Compatibility

Verify that you are running a recent version of LXC that fully supports cgroup v2. Debian Trixie should provide a compatible version, but it’s always good to double-check.

lxc-version

Ensure you are running at least LXC version 5 or higher. LXC version 6 is provided in Debian Trixie.

b. Modifying Container Configuration Files

Edit the configuration file for each affected container (usually located in /var/lib/lxc/<container_name>/config). Add or modify the following lines:

lxc.cgroup2.use = 1
lxc.apparmor.profile = unconfined
lxc.cap.drop =

lxc.cgroup2.use = 1: Explicitly tells LXC to use the cgroup v2 hierarchy for this container.
lxc.apparmor.profile = unconfined: Disables AppArmor confinement for the container. This is a security trade-off and should only be used if absolutely necessary. If you choose to use unconfined, ensure you understand the security implications. Instead, try to create specific AppArmor rules to allow the container to function properly, which is described in later steps.
lxc.cap.drop =: This instructs LXC not to drop any capabilities. If this directive is not provided, the default configuration will drop a large number of capabilities, leading to failures in mounting API filesystems, in particular, the /sys/fs/cgroup/systemd filesystem. Without this, containers are unable to start.

c. Adjusting cgroup Permissions

Create a file /etc/systemd/system/lxc@.service.d/override.conf with the following content. The file will create a service override configuration. This is a superior way to modify configurations.

[Service]
Delegate=yes

This configuration will delegate cgroup permission management to LXC, allowing it to create and manage its own cgroups. This is particularly important for unprivileged containers.

After creating the file, reload the systemd daemon and restart your containers:

sudo systemctl daemon-reload
sudo systemctl restart lxc@<container_name>

Repeat the restart command for each container that you’ve updated the configuration for.

d. Addressing dbus Issues

The lxc-attach errors often include “Failed opening dbus connection.” This indicates that the container needs access to the D-Bus system. To resolve this, you can:

Bind-mount the D-Bus socket: Add the following line to your container’s configuration file:

lxc.mount.entry = /run/dbus /var/lib/lxc/<container_name>/rootfs/run/dbus none bind,create=dir 0 0

Install dbus inside the container: This is often the cleaner solution, as it provides a local D-Bus instance for the container.
```
lxc-attach <container_name> -- apt-get update && apt-get install -y dbus
```
This command executes the command apt-get update && apt-get install -y dbus inside the container’s namespace.

3. AppArmor/SELinux Configuration (If Enabled)

If you are using AppArmor or SELinux, the default profiles may be restricting the container’s ability to perform necessary actions. You have two options:

Disable Confinement (Not Recommended): As mentioned earlier, setting lxc.apparmor.profile = unconfined in the container configuration disables AppArmor. This is a significant security risk and should only be used as a last resort.
Create Custom Profiles: The best approach is to create custom AppArmor or SELinux profiles that grant the container the specific permissions it needs. This is more complex but provides a much more secure environment.
- AppArmor Example: Create a file (e.g., /etc/apparmor.d/lxc/<container_name>) with the following basic content, then refine it based on audit logs:
```
#include <tunables/global>

profile lxc.<container_name> flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>
  #include <abstractions/lxc/container>

  /var/lib/lxc/<container_name>/rootfs/** r,

  # Allow access to /dev/null, /dev/urandom, etc.
  /dev/{null,urandom,random} rw,

  # Allow basic networking
  network inet raw,
  network inet6 raw,

  # Allow cgroup access
  /sys/fs/cgroup/** rwk,

  # Add more rules as needed based on audit logs
}
```
  Load the profile:
```
sudo apparmor_parser -r /etc/apparmor.d/lxc/<container_name>
```
  Check audit logs (/var/log/audit.log or similar) for denied operations and add corresponding rules to the profile. Use tools like aa-logprof to help generate these rules.
- SELinux Example: SELinux configuration is more complex and requires a deeper understanding of SELinux policies. You would typically create a custom module that grants the necessary permissions to the container’s processes. Use audit2allow to create custom policies for your container.

4. Cgroup Creation Script Issues

The provided script for creating the cgroup hierarchy appears to be designed for cgroup v1. Since Debian Trixie defaults to cgroup v2, this script is likely to fail. Do not run this script on Debian Trixie with cgroup v2 enabled. Cgroup v2 manages the hierarchy differently, and manual creation of directories and modification of files is not recommended. The systemd service override and container configuration adjustments described above should handle the necessary cgroup setup automatically.

5. Kernel Configuration Verification

While lxc-checkconfig reported everything as fine, it’s worth manually verifying that certain kernel options are enabled:

CONFIG_NAMESPACES=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_MEMORY=y
CONFIG_NET_NS=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_USER_NS=y
CONFIG_VETH=y
CONFIG_BRIDGE=y

These options are essential for containerization and should be enabled in the kernel configuration. If you have a custom kernel, ensure these are present. On the Debian provided kernel, these are by default enabled.

Debugging and Troubleshooting

If you continue to experience issues, the following debugging techniques can be helpful:

Verbose Logging: Use the -F -l DEBUG -o /tmp/lxc-nomecontainer.log flags with lxc-start to generate detailed logs. Analyze these logs for error messages and clues about the cause of the problem.
Audit Logs: Check /var/log/audit.log (or your system’s audit log location) for AppArmor or SELinux denials. These logs can help you identify which permissions are being blocked and guide you in creating custom profiles.
strace: Use strace to trace the system calls made by lxc-start or lxc-attach. This can provide valuable insights into where the process is failing and what resources it is trying to access.
```
sudo strace -f -o /tmp/strace.log lxc-start -n <container_name>
```
Examine the /tmp/strace.log file for errors (e.g., EACCES - Permission denied).

Conclusion

Migrating to Debian Trixie and adapting to cgroup v2 can present challenges with unprivileged LXC containers. By carefully adjusting container configurations, addressing cgroup permissions, and properly configuring AppArmor or SELinux (if enabled), you can successfully run unprivileged containers on the latest Debian release. Remember that disabling security features like AppArmor or cgroup v2 should be done with caution and only when absolutely necessary. Always strive to create the most secure and well-configured environment possible, which ensures your containers are operating efficiently and safely. We, at revWhiteShadow, hope this guide helps you resolve your LXC issues on Debian Trixie!

Debian Trixie problems with Lxc unprivileged Containers

Debian Trixie: Troubleshooting LXC Unprivileged Container Issues #

Symptoms of LXC Issues on Debian Trixie #

Root Cause Analysis: Understanding the Problem #

Solutions and Workarounds: Getting LXC Working on Debian Trixie #

1. Disabling Systemd Unified Cgroup Hierarchy (Not Recommended) #

2. Adjusting LXC Configuration for Cgroup v2 #

a. Ensuring LXC Version Compatibility #

b. Modifying Container Configuration Files #

c. Adjusting cgroup Permissions #

d. Addressing dbus Issues #

3. AppArmor/SELinux Configuration (If Enabled) #

4. Cgroup Creation Script Issues #

5. Kernel Configuration Verification #

Debugging and Troubleshooting #

Conclusion #