Navigating SSH Host Key Verification Failures: Understanding read_passphrase on Production Servers

At revWhiteShadow, we understand the critical role of secure and reliable automated deployments. It’s particularly frustrating when seemingly identical configurations across staging and production environments lead to divergent, unexplainable SSH errors. One such recurring issue that causes significant disruption is the perplexing scenario where the read_passphrase function is invoked on a production server during SSH operations, even when the SSH key is explicitly not protected by a password. This behavior, particularly when contrasted with a smoothly functioning staging environment, can be incredibly baffling. This comprehensive guide aims to demystify this specific SSH error, providing a deep dive into its potential causes and offering actionable solutions to restore seamless connectivity for your Bitbucket Pipelines and other automated tasks.

Understanding the Core of the Problem: SSH Host Key Verification

Before we delve into the specifics of read_passphrase, it’s crucial to grasp the fundamental security mechanism at play: SSH host key verification. When you initiate an SSH connection from a client to a server, the client needs to verify the identity of the server to prevent man-in-the-middle attacks. This verification is done by comparing the server’s host key against a record stored on the client, typically in the ~/.ssh/known_hosts file.

On your staging server, you observe:

debug1: Host '$STAGING_SERVER' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug1: ssh_rsa_verify: signature correct

This output clearly indicates that your client has previously connected to $STAGING_SERVER, its host key has been recorded in known_hosts, and the current connection’s key matches the stored one. Consequently, SSH proceeds without prompting for any further authentication related to the host itself.

However, on your production server, the scenario unfolds differently:

debug1: Host '$PRODUCTION_SERVER' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:5
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: permanently_drop_suid: 0
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Host key verification failed.

The presence of debug1: Host '$PRODUCTION_SERVER' is known and matches the RSA host key. and debug1: Found key in /root/.ssh/known_hosts:5 suggests that, like your staging server, the production server’s client also has a record of the production server’s host key. The anomaly arises with the subsequent lines, specifically debug1: read_passphrase: can't open /dev/tty: No such device or address and ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory.

The Mysterious read_passphrase and ssh-askpass

The read_passphrase function is integral to SSH’s authentication process, particularly when dealing with private keys that are protected by a passphrase. SSH needs a mechanism to prompt the user for this passphrase. In interactive sessions, this is typically handled by ssh-askpass, which opens a dialog box.

The error read_passphrase: can't open /dev/tty: No such device or address and ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory indicates that the SSH client on the production server is attempting to use ssh-askpass to retrieve a passphrase, but it cannot find or execute the necessary tool, or it cannot access the terminal (/dev/tty) to prompt for it.

This is the crux of the mystery: you’ve explicitly stated that your SSH key is not protected by a password. If the key has no passphrase, SSH should not be asking for one. So, why is read_passphrase being called?

Investigating the Root Cause: Beyond a Simple Passphrase Absence

The fact that read_passphrase is being called on your production server, despite your assurance that the SSH key itself is not password-protected, points to a more nuanced issue. It suggests that SSH is being tricked into believing that a passphrase is required, or that the SSH agent forwarding or the SSH client configuration is behaving differently in this specific environment.

#### SSH Agent and Key Management

While you might have created a new id_rsa key without a passphrase, the SSH client might still be interacting with the SSH agent (ssh-agent) or expecting a passphrase for reasons not immediately apparent from the key file itself.

  • SSH Agent: The ssh-agent is a background program that holds private keys used for public key authentication (like SSH). When you add a key to the agent using ssh-add, it decrypts the key (if it has a passphrase) and keeps it in memory. Subsequent SSH connections can then use the key from the agent without needing the passphrase again.
  • Potential Discrepancy: It’s possible that the SSH client on your production server is implicitly trying to use the SSH agent, and the agent is either not running, not configured correctly, or is somehow misinterpreting the key. When ssh-agent is used with an agent that does not have the key loaded, or if the key is supposed to be password protected and the agent doesn’t have it, SSH might fall back to attempting to prompt via ssh-askpass.

#### ssh_config and sshd_config Deep Dive

You mentioned that your ssh_config and sshd_config files are identical on both servers. This is excellent for ruling out basic configuration differences. However, even subtle variations or specific directives within these files can have unexpected consequences, especially when combined with the environment in which the SSH client is running (e.g., within a Bitbucket Pipeline runner).

Let’s consider what might be relevant:

  • IdentitiesOnly yes: In ssh_config, this directive tells SSH to only use identity files explicitly configured in the configuration or via the ssh-add command. If this is set to yes and your known_hosts file has an entry, but the actual key file path specified or implicitly used by the client doesn’t match what’s expected by the agent or the system, it could lead to issues.
  • PasswordAuthentication no: While you’re using key-based authentication, the absence of PasswordAuthentication no in the client’s ssh_config isn’t the direct cause of read_passphrase, but it’s good practice to ensure it’s configured correctly if password authentication is not desired.
  • ControlMaster and ControlPath: These directives allow you to multiplex multiple SSH sessions over a single network connection. While generally beneficial, misconfigurations here could theoretically lead to unexpected behavior in how SSH handles authentication state.
  • IdentityFile: If there’s an explicit IdentityFile directive pointing to a specific key, and that key is not correctly loaded or available to the SSH agent being used by the pipeline, it could trigger the read_passphrase behavior.

#### ssh-askpass Environment and Availability

The lines:

debug1: read_passphrase: can't open /dev/tty: No such device or address
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory

are particularly telling. They indicate that the SSH client is actively trying to execute ssh-askpass and expects it to be available to prompt for a passphrase.

  • Non-Interactive Environments: Bitbucket Pipelines, like many CI/CD runners, are non-interactive environments. They do not have a graphical terminal (/dev/tty) to display prompts.
  • Missing ssh-askpass: In some minimal containerized environments used for CI/CD, the ssh-askpass utility might not be installed by default. When SSH attempts to use it and it’s not found, you get the “No such file or directory” error.
  • SSH Agent Not Present or Accessible: If the SSH client in the pipeline cannot find or connect to a running ssh-agent that has your unprotected private key loaded, it might default to trying to prompt for a passphrase via ssh-askpass, even if the key itself has no passphrase. This could happen if the agent is not started or if the SSH_AUTH_SOCK environment variable is not set correctly.

#### Bitbucket Pipelines Specifics

Given that this is happening within Bitbucket Pipelines, we must consider how SSH keys are typically managed in such environments.

  • SSH Key Deployment: Bitbucket Pipelines provide a secure way to store and deploy SSH keys. You usually add your private SSH key as a repository variable (marked as “secure”). The pipeline runner then makes this key available to your build steps.
  • Key Permissions: When the private key is deployed, the pipeline runner usually sets appropriate permissions. However, if the key file itself is somehow treated as if it requires a passphrase (perhaps due to metadata or how it’s written to disk), this could be the trigger.
  • ssh-agent within Pipelines: It’s common practice in CI/CD to start an ssh-agent within the pipeline environment and add your private key to it. This is generally the most robust way to handle SSH authentication. If this step is missing or failing, SSH might fall back to less desirable methods.

Actionable Solutions: Resolving the read_passphrase Anomaly

Based on our analysis, the core issue likely stems from SSH incorrectly assuming a passphrase is required, or from the environment’s inability to handle interactive prompts or access an SSH agent properly. Here are the most effective strategies to resolve this:

## Solution 1: Explicitly Add Key to SSH Agent within Pipeline

This is the most recommended approach for CI/CD environments. It ensures that your key is loaded into an agent, which SSH clients can then query without needing interactive prompts or directly accessing the key file.

Steps:

  1. Ensure ssh-agent is Running: Before your SSH command, explicitly start ssh-agent and set the necessary environment variables.
  2. Add Your Private Key: Use ssh-add to add your private key to the running agent.

Example bitbucket-pipelines.yml snippet:

pipelines:
  default:
    - step:
        name: Deploy to Production
        script:
          # Start ssh-agent
          - eval "$(ssh-agent -s)"

          # Add your private SSH key (replace with your actual variable name)
          # Make sure you've added your private key as a secure repository variable named SSH_PRIVATE_KEY
          - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -

          # Set the correct permissions for the key (often handled by Bitbucket, but good to be explicit)
          # This might not be strictly necessary if using ssh-add with a variable, but good practice if saving to file.
          # - mkdir -p ~/.ssh
          # - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
          # - chmod 600 ~/.ssh/id_rsa

          # Now, run your SSH command. It should use the key from the agent.
          # Replace 'user@production-server' with your actual connection details.
          - ssh -o StrictHostKeyChecking=no user@production-server 'your_command_here'

Explanation:

  • eval "$(ssh-agent -s)": This command starts ssh-agent in the background and sets the environment variables (SSH_AUTH_SOCK and SSH_AGENT_PID) that other programs need to communicate with the agent.
  • echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -: This takes your private key (stored securely in Bitbucket as SSH_PRIVATE_KEY), removes any carriage returns that might cause issues, and pipes it to ssh-add -, which adds it to the agent. Crucially, since the key is not password-protected, ssh-add will not prompt for a passphrase.

Important Considerations:

  • Secure Variable: Ensure your private key is stored as a secure repository variable in Bitbucket (e.g., SSH_PRIVATE_KEY) and that its permissions within Bitbucket are set correctly.
  • tr -d '\r': This is vital if your key was generated or copied from a Windows environment, as CRLF line endings can cause ssh-add to fail or behave unexpectedly.
  • StrictHostKeyChecking=no: For initial setups or environments where you haven’t pre-populated known_hosts, you might need this. However, for production, it’s generally safer to manage known_hosts explicitly. You can fetch the production server’s host key beforehand and add it to your known_hosts file.

## Solution 2: Explicitly Specify Identity File (Less Recommended for Pipelines)

If for some reason ssh-agent is not feasible or you prefer to directly point SSH to the key file, you can use the IdentityFile directive in your ssh_config. However, this requires careful handling of the key file within the pipeline.

Steps:

  1. Create ~/.ssh/config: Create a configuration file.
  2. Add IdentityFile: Specify the path to your private key.
  3. Place Key File: Ensure the private key file is correctly placed and permissioned.

Example bitbucket-pipelines.yml snippet:

pipelines:
  default:
    - step:
        name: Deploy to Production
        script:
          # Create .ssh directory and config file
          - mkdir -p ~/.ssh
          - echo "Host production-server-alias" >> ~/.ssh/config
          - echo "  HostName production-server-ip-or-hostname" >> ~/.ssh/config
          - echo "  User your_ssh_user" >> ~/.ssh/config
          - echo "  IdentityFile ~/.ssh/id_rsa_prod" >> ~/.ssh/config
          - echo "  StrictHostKeyChecking yes" >> ~/.ssh/config # Or no, if managing known_hosts separately

          # Place your private key file
          - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa_prod
          - chmod 600 ~/.ssh/id_rsa_prod

          # Add production server's host key to known_hosts (highly recommended for security)
          # You would typically fetch this once and store it securely, or use ssh-keyscan
          # Example using ssh-keyscan (less secure if done every time):
          # - ssh-keyscan production-server-ip-or-hostname >> ~/.ssh/known_hosts
          # - chmod 644 ~/.ssh/known_hosts

          # Run your SSH command
          - ssh production-server-alias 'your_command_here'

Explanation:

  • This approach manually configures SSH to use a specific private key file (~/.ssh/id_rsa_prod) for a given host alias (production-server-alias).
  • It directly writes the private key from the secure variable to a file.
  • Security Risk: Directly writing private keys to disk in CI/CD environments, even temporarily, increases the attack surface. Using ssh-agent is generally preferred because the key is held in memory and not exposed as a file for extended periods.

## Solution 3: Check ssh-askpass Installation and Configuration (Less Likely for Key Absence)

While you’ve stated your key is not password protected, the fact that ssh-askpass is being invoked strongly suggests SSH is behaving as if it is. If you were to use this method (again, not recommended if your key is truly passphrase-free), you would ensure ssh-askpass is installed in your pipeline environment.

Steps:

  1. Install ssh-askpass: Add the installation command for your pipeline’s base image.
  2. Configure SSH_ASKPASS Environment Variable: Point SSH_ASKPASS to the installed utility.

Example bitbucket-pipelines.yml snippet (for demonstration, if ssh-agent fails):

pipelines:
  default:
    - step:
        name: Deploy to Production
        image: alpine:latest # Example image
        script:
          # Install ssh-askpass if available for the image (e.g., on Debian/Ubuntu based)
          # For Alpine, sshpass might be used differently, or you'd rely on ssh-agent
          # Example for Debian/Ubuntu:
          # - apt-get update && apt-get install -y ssh-askpass

          # On Alpine, ssh-client might include it or it needs to be provided.
          # A more common Alpine approach is 'apk add sshpass' and using that,
          # but this is for understanding ssh-askpass.
          # If your image doesn't have it, and you need it (unlikely for no-passphrase keys),
          # you'd need to build a custom image or find an alternative.

          # If using ssh-askpass, you might need to set this,
          # but this is for interactive sessions, which pipelines are not.
          # - export SSH_ASKPASS=/usr/bin/ssh-askpass
          # - export DISPLAY=:0 # Not applicable in pipelines

          # ... proceed with other SSH commands ...

Why this is unlikely to be the primary solution here:

  • Pipelines are non-interactive. ssh-askpass is designed for interactive sessions.
  • Your key is not password protected. SSH should not need to prompt for a passphrase. The invocation of ssh-askpass is likely a symptom of another underlying issue, not the cause itself.

## Solution 4: Verifying known_hosts Entries and Key Formats

While you mentioned the host key is found and matches, it’s worth double-checking the integrity of the known_hosts file and the exact format of the keys.

  • Corrupted Entries: A corrupted line in known_hosts could cause SSH to re-verify or behave unexpectedly.
  • Key Type Mismatch: Although you mentioned RSA, ensure there isn’t a mix of key types or an outdated entry in known_hosts that conflicts with the current server key.
  • Permissions on known_hosts: Ensure the user running the SSH command has read access to ~/.ssh/known_hosts.

To diagnose:

  • Remove and Re-add: Temporarily remove the production server’s entry from ~/.ssh/known_hosts on the client side (or within your pipeline’s SSH configuration) and let SSH re-populate it on the first connection.
  • Use ssh-keyscan: In your pipeline setup, you can use ssh-keyscan production-server-ip-or-hostname >> ~/.ssh/known_hosts to explicitly add the host key. Be mindful of security implications if you’re not validating the key fingerprint manually.

## Solution 5: Examining the SSH Client Version and Environment

Subtle differences in the SSH client version or the underlying operating system of the pipeline runners could also play a role. While less common, an older SSH client might have bugs or different default behaviors.

  • Client Version: Check the ssh -V output on both your staging and production environments (if you can access their clients) and compare it with the version available in your Bitbucket Pipeline runner’s image.
  • Environment Variables: Ensure no other environment variables are interfering with SSH’s behavior.

Final Thoughts and Best Practices

The error read_passphrase: can't open /dev/tty: No such device or address when your SSH key is not password-protected is a strong indicator that SSH is incorrectly assuming a passphrase is required, or that the mechanism it relies on to obtain a passphrase (like an SSH agent or ssh-askpass) is unavailable or misconfigured in the non-interactive pipeline environment.

At revWhiteShadow, our experience confirms that managing SSH keys via an SSH agent within your CI/CD pipeline is the most robust and secure method. It abstracts away the complexities of direct key file handling and avoids the pitfalls of non-interactive environments attempting to solicit interactive input.

By diligently implementing the ssh-agent solution (Solution 1), you should be able to bypass the read_passphrase issue and achieve the seamless SSH connectivity required for your production deployments. Always prioritize security by using secure repository variables and managing known_hosts entries to prevent man-in-the-middle attacks. This detailed approach should empower you to outrank any resource focusing solely on a superficial understanding of this complex SSH error.