regreSSHion: Code Execution Vulnerability in OpenSSH server (CVE-2024-6387)

The Qualys Threat Research Unit (TRU) has discovered a Remote Unauthenticated Code Execution (RCE) vulnerability in OpenSSH’s server (sshd) on glibc-based Linux systems. This vulnerability has been assigned CVE-2024-6387.

The vulnerability, caused by a signal handler race condition in OpenSSH’s server (sshd), allows unauthenticated remote code execution (RCE) as root on glibc-based Linux systems, presenting a significant security risk. This race condition affects sshd in its default configuration.

The Bitnami catalog is based on Debian, according to the Debian security tracker:

• Debian 11 (bullseye) is not affected.

• Debian 12 (bookworm) is affected up to version 1:9.2p1-2+deb12u3.

SSH server is installed and running in OVAs and Cloud Images for AWS, Google, and Azure Marketplaces. Bitnami Helm charts and container images are not affected. The Bitnami team is working on releasing new versions in all the Marketplaces.

See below some details about how the bundled SSH package can be upgraded to a patched version:

Fix/Mitigation

By default, OVAs and Cloud Images include the unattended-upgrades package that will try to install security updates automatically daily. However, it is possible to force the execution of the cronjob manually.

First of all, verify you are running an affected version of the openssh package as shown below

$ sudo dpkg -l | grep ssh

ii  libssh2-1:amd64                  1.10.0-3+b1                    amd64        SSH2 client-side library

ii  openssh-client                   1:9.2p1-2+deb12u2              amd64        secure shell (SSH) client, for secure access to remote machines

ii  openssh-server                   1:9.2p1-2+deb12u2              amd64        secure shell (SSH) server, for secure access from remote machines

ii  openssh-sftp-server              1:9.2p1-2+deb12u2              amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines

ii  ssh                              1:9.2p1-2+deb12u2              all          secure shell client and server (metapackage)

In case you are affected, force the unattended-upgrade execution by running the command below

$ sudo apt-get update && sudo unattended-upgrade -d

This will log new information into the /var/log/unattended-upgrades/unattended-upgrades.log and /var/log/unattended-upgrades/unattended-upgrades-dpkg.log files, where you can check if the OpenSSH service has been updated and the new version it has installed

$ grep -i ssh /var/log/unattended-upgrades/unattended-upgrades-dpkg.log

Preparing to unpack .../1-openssh-sftp-server_1%3a9.2p1-2+deb12u3_amd64.deb ...

Unpacking openssh-sftp-server (1:9.2p1-2+deb12u3) over (1:9.2p1-2+deb12u2) ...

Preparing to unpack .../2-openssh-server_1%3a9.2p1-2+deb12u3_amd64.deb ...

Unpacking openssh-server (1:9.2p1-2+deb12u3) over (1:9.2p1-2+deb12u2) ...

Preparing to unpack .../3-openssh-client_1%3a9.2p1-2+deb12u3_amd64.deb ...

Unpacking openssh-client (1:9.2p1-2+deb12u3) over (1:9.2p1-2+deb12u2) ...

Preparing to unpack .../5-ssh_1%3a9.2p1-2+deb12u3_all.deb ...

Unpacking ssh (1:9.2p1-2+deb12u3) over (1:9.2p1-2+deb12u2) ...

Setting up openssh-client (1:9.2p1-2+deb12u3) ...

Setting up openssh-sftp-server (1:9.2p1-2+deb12u3) ...

Setting up openssh-server (1:9.2p1-2+deb12u3) ...

After that, you can check the new version has been installed

$ sudo dpkg -l | grep ssh

ii  libssh2-1:amd64                  1.10.0-3+b1                    amd64        SSH2 client-side library

ii  openssh-client                   1:9.2p1-2+deb12u3              amd64        secure shell (SSH) client...

ii  openssh-server                   1:9.2p1-2+deb12u3              amd64        secure shell (SSH) server...

ii  openssh-sftp-server              1:9.2p1-2+deb12u3              amd64        secure shell (SSH) sftp...

ii  ssh                              1:9.2p1-2+deb12u3              all          secure shell client and server (metapackage)

From the client side you can check the server is returning the updated package information by running the next command

$ ssh -v <user>@<ip-address> 2>&1 | grep -i openssh

OpenSSH_9.6p1, LibreSSL 3.3.6

debug1: Local version string SSH-2.0-OpenSSH_9.6

debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u3

debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u3 pat OpenSSH* compat 0x04000000

If you have any questions about this process, please create an issue in our GitHub repository. We will be happy to help!

Updates

• [July 13, 2024, 10:05 AM (UTC)]:
• 130 out of 132 (98%) OVAs released
• 131 out of 133 (98%) AWS Images released
• 79 out of 81 (98%) Azure Images released
• 83 out of 84 (99%) Google Images released
• [July 11, 2024, 05:37 AM (UTC)]:
• 130 out of 132 (98%) OVAs released
• 131 out of 133 (98%) AWS Images released
• 78 out of 81 (96%) Azure Images released
• 82 out of 84 (98%) Google Images released
• [July 9, 2024, 06:12 AM (UTC)]:
• 129 out of 132 (98%) OVAs released
• 130 out of 133 (98%) AWS Images released
• 77 out of 81 (95%) Azure Images released
• 82 out of 84 (98%) Google Images released
• [July 3, 2024, 11:30 AM (UTC)]:
• 129 out of 132 (98%) OVAs released
• 129 out of 133 (98%) AWS Images released
• 76 out of 81 (94%) Azure Images released
• 76 out of 84 (91%) Google Images released

Enhancing the Bitnami Helm Charts Experience: Changelog, tags, and validation images

Bitnami has recently rolled out several initiatives aimed at enhancing the user experience with Helm charts. These improvements focus on better traceability and smoother integrations. Read on to discover the latest updates:

Improved Changelog and Tagging System

One major initiative is the enhancement of change traceability for Bitnami Helm charts. This has been achieved by introducing a CHANGELOG.md file for every Helm chart and implementing git tags for every new version.

Automated Changelog Updates

With every pull request (PR) merge a new CHANGELOG.md file is automatically updated to list the changes included in that specific release. This automation is powered by the conventional-changelogs-cli, eliminating the need for contributors to perform this step manually.

For example, here is what a typical CHANGELOG.md file looks like. These files are excluded from the Helm charts themselves, as specified in the .helmignore file. Starting today, you will see CHANGELOG.md files gradually being rolled out to every Bitnami Helm chart as new releases are produced.

Consistent Version Tagging

In addition to the changelog updates, every chart change now results in a commit tag formatted as “APP/VERSION”. An example of such a tag can be seen here: spark/9.0.4.

These enhancements are designed to assist users during the upgrade process and improve compatibility with automation tools like Renovate and GitHub Dependabot.

In the following example, we have a Helm chart (Airflow) with three dependencies: bitnami/redis, bitnami/postgresql, and bitnami/common. We will use Renovate to automatically detect and create Pull Requests every time there is a new version of these dependencies.

apiVersion: v2

appVersion: 2.9.1

dependencies:

- condition: redis.enabled

  name: redis

  repository: oci://registry-1.docker.io/bitnamicharts

  version: 19.2.0

- condition: postgresql.enabled

  name: postgresql

  repository: oci://registry-1.docker.io/bitnamicharts

  version: 15.2.0

- name: common

  repository: oci://registry-1.docker.io/bitnamicharts

  version: 2.19.3

name: airflow

version: 18.1.1

Following the official Renovate installation instructions we enabled the automation in the repository

In the automated PR, we can see that it detected our helm chart:

Once this is merged, after some time we will see PRs like the following:

Checking its contents, we can see that the changelog is included in the PR description:

Warning on Replacing Default Images

We have introduced a new validation that displays warnings when default images bundled in the Helm chart are replaced. This is to let users know when the images from a Helm chart have been altered

Each Helm chart is meticulously designed, tested, and validated using a specific set of Bitnami container images across multiple platforms. Replacing these default containers can introduce several risks:

• Degraded Security and Performance: Non-Bitnami containers may not have the same security features and optimizations, leading to potential vulnerabilities and performance issues.

• Broken Chart Features: The Helm chart’s functionality might rely on specific configurations or tools available only in the original Bitnami containers.

• Missing Environment Variables: Substituted containers may lack critical environment variables necessary for the Helm chart to function correctly.

• Security: A malicious threat actor could have switched the container images and redistributed the artifact as a legit Bitnami Helm chart.

When deploying a Helm chart, if the images that Bitnami has built the Helm chart with are replaced, a warning will appear in the console to alert the user of these potential risks. We understand that some users might need to switch the container images that Bitnami has verified, but at the same time, we believe making users aware of this change is important for the reasons above.

Branch size reduction

Bitnami has recently reduced the size of certain branches related to index.yaml, as outlined in this GitHub Issue. Previously, Helm charts were distributed using the index.yaml method, which has since been replaced by OCI through DockerHub. You can find the OCI Helm charts here.

Despite the shift to OCI, index.yaml was maintained for backward compatibility. However, the sheer number of releases and commits generated by our automated test and release pipeline caused these branches to balloon in size:

• index: 2.23 GiB

• archive-full-index: 987.42 MiB

This significant size increase resulted in longer clone times and made life difficult for those users looking to contribute fixes or improvements.

To address this issue, we implemented automation to squash all commits in the index-related branches. This drastic size reduction has yielded the following results:

• index: 840.41 KiB

• archive-full-index: 1.89 MiB

These changes significantly improve the contribution experience, making it easier and faster for our community to collaborate and contribute.

If you want to use Bitnami packages in production environments for mission-critical use cases, check out Tanzu Application Catalog—an enterprise version of Bitnami with several exclusive features that include base OS customization, app-level customization, Vulnerability Exploitability eXchange (VEX), SBOM, SLSA L3, and more.

Bitnami Helm Charts are Now More Secure Than Ever

Bitnami-packaged open source software is loved by developers for its ease of use, which enables developers to directly pull a Bitnami package and seamlessly start using it with little effort. The fact that Bitnami-packaged open source software accounts for over 3 billion pulls per year on DockerHub is a testament to its popularity among developers. But, apart from the ease of use, we also aim to make our software inherently more secure and reliable by updating packaging practices per industry standards. That’s why, over the past few weeks, our team has worked on improving the security of Bitnami-packaged Helm charts.

As a starting point for these improvements, we decided to leverage the risk analysis, security compliance, and misconfiguration scanning capabilities of Armo’s Kubescape (an open source Kubernetes security platform). The best practices codified as controls by Armo helped us identify a list of preventative, detective, or corrective measures, which we then implemented. Some of the major improvements we have completed are listed below.

Enabling containers to function without group root

Until now, Bitnami charts used user 1001 and group root (following Openshift standards). However, Kubernetes platforms increasingly encourage the use of more secure configurations for applications, which does not allow for the use of the group root. With the new changes, Bitnami containers and Helm charts will no longer need the group root to function, making 1001 the new default runAsGroup. This setting can be reverted by setting runAsGroup back to 0. 

Enabling usage of immutable filesystems

Using immutable filesystems is a mandatory requirement of security checklists such as NSA or MITRE. This configuration helps enforce an immutable infrastructure strategy; the container only needs to write on the mounted volume that persists in the state. An immutable root filesystem is also capable of preventing malicious binaries from writing to the host system. To enable the use of immutable filesystems, we changed all writable paths of containers to emptyDir volumes. In some cases, we needed to add extra init containers that copy folders like conf, plugins, or logs. This is necessary because mounted volumes replace the contents of the original folders. It’s important to note that this may cause issues with customization scripts, such as initScripts or the use of custom commands and arg. Using older versions of Bitnami containers may cause problems because they might lack the necessary bash changes to allow readOnlyRootFilesystem. This setting can be reverted by setting readOnlyRootFilesystem back to false.

Ensuring no pod is without a resource request or limit

Having pods without any resource requests or limits is increasingly discouraged as these containers may deplete node resources. Currently, our charts warn users if the resources object is not set. Additionally, we have the resourcesPreset value to assist users when testing different resource configurations, but this is not recommended for production. With the new changes, we set the resourcesPreset value to the minimum size that operates in our internal testing. This size is not the recommended size by the Bitnami team, but it is the minimum size for basic testing of the solution. Users must set resource values according to their requirements and use cases. 

It’s important to note that users already setting the resources value will not be affected by this change. Users who are not setting resources may experience a performance drop in their workloads, as they would be configured to a minimal value for basic testing. In these cases, we strongly recommend users set the resources value before any upgrade. This setting can be reverted by setting resourcesPreset back to none.

Enabling functioning in OpenShift restricted-v2 SCC

Before our changes, Bitnami charts did not function in Openshift restricted-v2 Security Context Constraints(SCC) because they set user and group to 1001. To address this, we enabled the automatic adaptation of the containerSecurityContext and podSecurityContext sections when running in Openshift installations. If the detected platform is Openshift, the values runAsUser, runAsGroup, and fsGroup will be automatically removed, allowing the Openshift platform to select the appropriate user and group IDs.

Users deploying in platforms other than Openshift are not affected by this change. Openshift users who set their runAsUser, runAsGroup, and fsGroup values will find that these are removed during an upgrade. This setting can be reverted by setting global.compatibility.openshift.adaptSecurityContext to false.

For detailed information on all improvements check out this GitHub issue.

Using the Tanzu OSS Health Assessment to understand the impact of these changes

Earlier this month, we launched the Tanzu OSS Health Assessment—a free-to-use tool that provides a comprehensive overview of your open source software dependencies and the potential risks you may face. 

To understand how the new improvements in Bitnami packages can impact your security posture, let’s look at the OSS health assessment report generated before and after the security improvements were implemented.

Let’s look at the OSS health assessment report of a repository with the Helm charts—Nginx, ClickHouse, Grafana and PostgreSQL—before the security improvements were made.

As you can see, the percentage of Misconfigured resources was 54.55%.

Now let’s look at the health assessment report of the same Helm charts after the implementation of our improvement measures.

From the snaps, you can see that misconfigured resources have dropped from 54.55% to 17.39%, implying a significantly better security posture, thanks to the aforementioned improvements.

You can also see that the security vulnerabilities plummeted from 684 to just 4. This drop is because the Bitnami Helm charts are sourced from the Tanzu Application Catalog with Photon OS as the base image, and not the standard Debian OS (which is the base OS for the community edition of Bitnami packages).

Thus, the recent security improvements in Bitnami-packaged Helm charts combined with Tanzu Application Catalog’s capability to let you use the VMware by Broadcom-maintained Linux distro—Photon OS as the base image can help you improve your security posture and minimize risks from your open source software dependencies.

Next Steps

To solve problems you may have with the Bitnami community packages—including deployment support, operational support, and bug fixes—please open an issue in the Bitnami Helm charts or containers GitHub repository. You can get the latest Bitnami-packaged software from our website, DockerHub (containers and Helm charts), Google Marketplace, Azure Marketplace, or AWS Cloud Marketplace. If you want to use Bitnami packages in production environments for mission-critical use cases, check out Tanzu Application Catalog—an enterprise version of Bitnami Application Catalog with several exclusive features that include base OS customization, app-level customization, Vulnerability Exploitability eXchange (VEX), and more. Take our OSS health assessment today and kickstart your journey toward the optimization of your open source software dependencies.