Mastering less +F on Ubuntu 22.04: Ensuring Persistence When Log Files Vanish

At revWhiteShadow, we understand the critical need for uninterrupted log monitoring, especially in dynamic server environments. One of the most common challenges faced by system administrators and developers on Ubuntu 22.04, when utilizing the powerful less pager with the +F option for following log files, is its tendency to exit automatically when the tailed file disappears. This behavior, often encountered during log rotation events or temporary file system issues, can disrupt crucial real-time insights into application and system behavior. We aim to provide a comprehensive solution that ensures less +F remains active, retaining its display and resuming its tailing function when the log file is recreated.

The Challenge: less +F and Disappearing Log Files

The standard less +F command is an invaluable tool for observing log files as they are updated. The +F option specifically instructs less to follow the end of the file, much like tail -f. However, the fundamental issue arises when the target log file is temporarily removed or renamed, as commonly occurs during log rotation procedures. When this happens, less +F typically terminates, displaying an error message indicating that the file no longer exists. While this is a predictable behavior, it is often an undesirable outcome for those needing continuous visibility.

Consider a scenario where a program, let’s say xyz, is actively writing to /var/log/xyz.log. You are diligently monitoring this file using less +F /var/log/xyz.log. The output scrolls by, showing lines xyz output line 101, xyz output line 102, and xyz output line 103. Suddenly, logrotate is triggered. This process typically renames the current log file (e.g., to /var/log/xyz.log.1) and creates a new, empty /var/log/xyz.log file to which the application will write. In this transition, the original /var/log/xyz.log briefly ceases to exist. The standard less +F session would likely terminate at this point. Subsequently, when xyz begins writing to the newly created /var/log/xyz.log, starting with xyz output line 104 and xyz output line 105, you would have to manually restart less +F to resume monitoring. This interruption breaks the continuity of observation and can lead to missed critical log entries during the restart phase.

The --follow-name option in less is often suggested as a solution for this problem. Its intention is to follow the filename rather than the file descriptor, meaning it should adapt if the file is recreated with the same name. However, our experience, and the experience of many users, confirms that even with --follow-name, less can still exit when the file disappears. The underlying mechanism of less often still relies on an open file descriptor, and when that descriptor points to a non-existent file, the program terminates. This is the precise pain point we are addressing: the need for less +F to be resilient to temporary file disappearances.

Exploring Workarounds and Their Limitations

Several approaches can be employed to mitigate the abrupt exit of less +F. One common method involves using a wrapper script to automatically restart the less +F process when it terminates. A basic example of such a script might look like this:

#!/bin/bash
while true
do
  less +F "${@}"
  rc=$?
  # Exit if less was exited cleanly (return code 0)
  # or if it was aborted by a signal (return code 128+signal_number)
  # For our purpose, we only want to re-run if the file disappeared,
  # which typically results in a specific error code like 1 in some configurations.
  # We will refine this logic.
  if [ ${rc} -eq 0 ] || [[ ${rc} -gt 128 ]]; then
    break
  fi
  echo "Log file disappeared or an error occurred. Restarting less +F in 5 seconds..."
  sleep 5
done
exit ${rc}

While this script can successfully restart less +F, it suffers from two significant drawbacks:

Flaw #1: Screen Clearing and Loss of Historical Context

The primary issue with this wrapper script is that each time less +F is restarted, the terminal screen is cleared, and all the previously displayed log content is lost. The user is presented with a blank slate, and the essential historical context of the log messages preceding the file’s disappearance is gone. This defeats a major advantage of using less, which is its ability to scroll back and examine past output. Ideally, we want the previously viewed lines to remain visible, providing a seamless transition from the moment before the file vanished to the moment it reappears and new data is written.

Flaw #2: Indiscriminate Restarts

The second flaw lies in the lack of specificity in the restart condition. The simple wrapper script shown above would attempt to restart less +F for any non-zero exit code, not just those specifically caused by the file’s disappearance. There are various other reasons why less +F might exit with an error. For instance, permission issues, encountering unreadable characters, or other unexpected conditions could lead to termination. We only desire an automatic restart when the specific condition of the tailed file vanishing occurs. Restarting in other error scenarios might mask underlying problems or lead to an endless loop of restarts if the root cause isn’t addressed.

The tail -f --retry Alternative: What’s Missing?

A common alternative suggested for this scenario is the tail -f --retry command. The --retry option in tail is designed to continue retrying to open a file if it becomes inaccessible. This indeed addresses the issue of the file disappearing and reappearing. When logrotate renames the file, tail -f will pause its operation, and upon the recreation of the log file, it will resume tailing.

However, tail -f --retry lacks the interactive capabilities that make less so powerful. The primary limitation is the inability to easily interrupt the tailing process to search, scroll, and analyze the output. With less +F, you can press Ctrl+C to temporarily halt the tailing, enter search commands (like / for forward search or ? for backward search), navigate through the historical log data using arrow keys, Page Up, and Page Down, and then easily resume tailing by simply pressing F. This dynamic switching between real-time monitoring and historical analysis is a cornerstone of effective log inspection. tail -f --retry does not offer this ability. Once you are in tail -f mode, the only way to interact with the data is by stopping the command and re-running it, or by piping the output to another tool.

The Quest for the Ideal Solution: less +F with Persistent View and Smart Retries

Our goal is to achieve a less +F experience that is robust against log file disappearances while retaining its core interactive functionalities. This means:

  1. Preserving the displayed output: When the log file is temporarily gone, the lines less has already displayed should remain visible.
  2. Intelligent retries: less should only attempt to re-tail the file if it specifically detects that the file has disappeared and has been recreated. Other error conditions should not trigger an automatic restart.
  3. Seamless resumption: Upon recreation of the log file, less +F should automatically resume tailing from where it left off, appending new entries to the preserved historical view.
  4. Full less interactivity: All the familiar less commands for searching, scrolling, and navigating should remain fully functional throughout this process.

Given these stringent requirements, we need to explore solutions that go beyond simple wrappers or alternative commands that sacrifice functionality.

Deep Dive into less Configuration and Potential Enhancements

The less utility is highly configurable, and it’s worth examining its options and behaviors more closely to see if the desired functionality can be achieved through its existing features or subtle modifications.

The +F option is essentially a shortcut for less -fF. The -f option tells less to not paginate, but instead to “follow” the file like tail -f. The capital -F (or +F) is the key here, indicating that less should automatically enter the “follow” mode when the file is first opened, meaning it will immediately start displaying new data as it arrives.

The behavior when the file disappears is often tied to how less manages file descriptors and internal state. When a file is deleted, the operating system invalidates the file descriptor that less holds for that file. less then typically detects this invalid state and exits. The --follow-name option, as mentioned, attempts to mitigate this by monitoring the filename. If the file is deleted and a new file with the same name is created, --follow-name should theoretically allow less to reopen it. However, as observed, this is not always consistently reliable for preventing exits.

Investigating less Source Code for Customization

For users with advanced technical capabilities and a need for precise control, modifying the less source code is a potential avenue. This would involve understanding less’s internal handling of file I/O, specifically how it detects file disappearance and how it manages the --follow-name logic.

A skilled C programmer might look into the main.c and file.c (or similar) files within the less source code. The key areas to examine would be:

  • File descriptor management: How less opens, reads from, and monitors file descriptors.
  • stat() and fstat() calls: These system calls are used to get file status information. Detecting a change in file inode or a non-existent file often involves comparing results from successive stat() calls or checking for specific error codes (like ENOENT for “No such file or directory”).
  • The --follow-name implementation: Understanding how this option hooks into the file monitoring loop. It likely involves periodically checking if the file exists and matches the expected inode, or if the filename has been re-associated with a new file.
  • Error handling: Identifying the specific error codes that less returns when a file disappears and how these are propagated to the main loop.

A potential modification could involve:

  1. Enhancing --follow-name: Making the logic more robust to detect when a file has been deleted and then recreated with the same name. This might involve more frequent stat() checks on the filename and comparing inode numbers.
  2. Introducing a new option: A new command-line flag, for instance, --persist-on-disappear, could be introduced. When this option is active, upon detecting a file disappearance (e.g., ENOENT error from stat), less would not exit. Instead, it would enter a dormant state, keeping its current buffer visible. It would then periodically re-check for the existence of the file. Upon successful re-creation, it would automatically resume the +F tailing operation. This would require careful management of the display and input handling to remain responsive while waiting.

Such modifications are complex and require a thorough understanding of Unix system programming and less’s internal architecture. For most users, this is likely not a practical solution.

The Search for a Dedicated Utility

Given the complexity of modifying less and the limitations of existing workarounds, the question arises: is there another utility that natively provides this exact functionality?

Our extensive research into various log tailing utilities, including popular Python and Ruby scripts found on platforms like GitHub, has revealed many excellent tools. Some offer advanced filtering, colorization, remote tailing, and sophisticated error handling. However, none that we have encountered explicitly provide the combination of:

  • Non-exiting tailing upon file disappearance.
  • Preservation of the historical screen buffer.
  • Intelligent resumption of tailing upon file recreation.
  • Seamless integration with interactive less-like navigation (interrupt, search, resume).

Many tools that aim for robustness often resort to strategies like automatically restarting the process or simply stopping when the file is gone, lacking the nuanced behavior we are seeking. The power of less lies in its ability to allow the user to pause tailing at any moment, perform complex searches and analysis on the previously displayed data, and then resume tailing with a simple keypress. This interactive loop is hard to replicate.

Conclusion: Achieving Resilient Log Monitoring on Ubuntu 22.04

While the exact functionality of a less +F that never exits upon file disappearance, persists its view, and intelligently resumes tailing is not natively available through standard less options or easily replicated with simple wrappers, understanding the underlying mechanisms provides clarity.

The ideal scenario, where less +F behaves like a tail -f --retry but with the added benefit of less’s interactive navigation, remains an advanced requirement. For most users on Ubuntu 22.04 seeking to improve their log monitoring experience in the face of log rotation, the closest practical approach without source code modification often involves a trade-off.

If the interactive scrolling and searching capabilities of less are paramount, one might need to accept the brief interruption and manual restart after log rotation, or accept the limitations of a wrapper script that clears the screen on restart. If continuous, uninterrupted tailing is the absolute priority, tail -f --retry is a viable option, albeit without the rich interactive analysis features of less.

The challenge highlights a niche in log monitoring tools, where the seamless integration of real-time following with deep historical inspection, coupled with resilience to file system events, is a highly sought-after but rarely perfectly realized feature. At revWhiteShadow, we continue to advocate for tools and practices that empower users with the best possible insight into their systems, and the pursuit of such robust logging solutions is an ongoing endeavor. For those willing to delve into the intricacies of system programming, the less source code offers the ultimate canvas for implementing such advanced, resilient tailing capabilities.