Wednesday, May 22, 2024

Enhancing the Bitnami Helm Charts Experience: Changelog, tags, and validation images

Bitnami has recently rolled out several initiatives aimed at enhancing the user experience with Helm charts. These improvements focus on better traceability and smoother integrations. Read on to discover the latest updates:

Improved Changelog and Tagging System

One major initiative is the enhancement of change traceability for Bitnami Helm charts. This has been achieved by introducing a CHANGELOG.md file for every Helm chart and implementing git tags for every new version.

Automated Changelog Updates

With every pull request (PR) merge a new CHANGELOG.md file is automatically updated to list the changes included in that specific release. This automation is powered by the conventional-changelogs-cli, eliminating the need for contributors to perform this step manually.


For example, here is what a typical CHANGELOG.md file looks like. These files are excluded from the Helm charts themselves, as specified in the .helmignore file. Starting today, you will see CHANGELOG.md files gradually being rolled out to every Bitnami Helm chart as new releases are produced.

Consistent Version Tagging

In addition to the changelog updates, every chart change now results in a commit tag formatted as “APP/VERSION”. An example of such a tag can be seen here: spark/9.0.4.


These enhancements are designed to assist users during the upgrade process and improve compatibility with automation tools like Renovate and GitHub Dependabot.


In the following example, we have a Helm chart (Airflow) with three dependencies: bitnami/redis, bitnami/postgresql, and bitnami/common. We will use Renovate to automatically detect and create Pull Requests every time there is a new version of these dependencies.

apiVersion: v2

appVersion: 2.9.1

dependencies:

- condition: redis.enabled

  name: redis

  repository: oci://registry-1.docker.io/bitnamicharts

  version: 19.2.0

- condition: postgresql.enabled

  name: postgresql

  repository: oci://registry-1.docker.io/bitnamicharts

  version: 15.2.0

- name: common

  repository: oci://registry-1.docker.io/bitnamicharts

  version: 2.19.3

name: airflow

version: 18.1.1


Following the official Renovate installation instructions we enabled the automation in the repository

In the automated PR, we can see that it detected our helm chart:

Once this is merged, after some time we will see PRs like the following:

Checking its contents, we can see that the changelog is included in the PR description:


Warning on Replacing Default Images
We have introduced a new validation that displays warnings when default images bundled in the Helm chart are replaced. This is to let users know when the images from a Helm chart have been altered

Each Helm chart is meticulously designed, tested, and validated using a specific set of Bitnami container images across multiple platforms. Replacing these default containers can introduce several risks:


  • Degraded Security and Performance: Non-Bitnami containers may not have the same security features and optimizations, leading to potential vulnerabilities and performance issues.

  • Broken Chart Features: The Helm chart’s functionality might rely on specific configurations or tools available only in the original Bitnami containers.

  • Missing Environment Variables: Substituted containers may lack critical environment variables necessary for the Helm chart to function correctly.

  • Security: A malicious threat actor could have switched the container images and redistributed the artifact as a legit Bitnami Helm chart.


When deploying a Helm chart, if the images that Bitnami has built the Helm chart with are replaced, a warning will appear in the console to alert the user of these potential risks. We understand that some users might need to switch the container images that Bitnami has verified, but at the same time, we believe making users aware of this change is important for the reasons above.


Branch size reduction


Bitnami has recently reduced the size of certain branches related to index.yaml, as outlined in this GitHub Issue. Previously, Helm charts were distributed using the index.yaml method, which has since been replaced by OCI through DockerHub. You can find the OCI Helm charts here.


Despite the shift to OCI, index.yaml was maintained for backward compatibility. However, the sheer number of releases and commits generated by our automated test and release pipeline caused these branches to balloon in size:

  • index: 2.23 GiB

  • archive-full-index: 987.42 MiB


This significant size increase resulted in longer clone times and made life difficult for those users looking to contribute fixes or improvements.


To address this issue, we implemented automation to squash all commits in the index-related branches. This drastic size reduction has yielded the following results:

  • index: 840.41 KiB

  • archive-full-index: 1.89 MiB


These changes significantly improve the contribution experience, making it easier and faster for our community to collaborate and contribute.


If you want to use Bitnami packages in production environments for mission-critical use cases, check out Tanzu Application Catalog—an enterprise version of Bitnami with several exclusive features that include base OS customization, app-level customization, Vulnerability Exploitability eXchange (VEX), SBOM, SLSA L3, and more.


Wednesday, April 17, 2024

Bitnami Helm Charts are Now More Secure Than Ever

Bitnami-packaged open source software is loved by developers for its ease of use, which enables developers to directly pull a Bitnami package and seamlessly start using it with little effort. The fact that Bitnami-packaged open source software accounts for over 3 billion pulls per year on DockerHub is a testament to its popularity among developers. But, apart from the ease of use, we also aim to make our software inherently more secure and reliable by updating packaging practices per industry standards. That’s why, over the past few weeks, our team has worked on improving the security of Bitnami-packaged Helm charts.

As a starting point for these improvements, we decided to leverage the risk analysis, security compliance, and misconfiguration scanning capabilities of Armo’s Kubescape (an open source Kubernetes security platform). The best practices codified as controls by Armo helped us identify a list of preventative, detective, or corrective measures, which we then implemented. Some of the major improvements we have completed are listed below.

Enabling containers to function without group root

Until now, Bitnami charts used user 1001 and group root (following Openshift standards). However, Kubernetes platforms increasingly encourage the use of more secure configurations for applications, which does not allow for the use of the group root. With the new changes, Bitnami containers and Helm charts will no longer need the group root to function, making 1001 the new default runAsGroup. This setting can be reverted by setting runAsGroup back to 0. 

Enabling usage of immutable filesystems

Using immutable filesystems is a mandatory requirement of security checklists such as NSA or MITRE. This configuration helps enforce an immutable infrastructure strategy; the container only needs to write on the mounted volume that persists the state. An immutable root filesystem is also capable of preventing malicious binaries from writing to the host system. To enable use of immutable filesystems, we changed all writable paths of containers to emptyDir volumes. In some cases, we needed to add extra init containers that copy folders like conf, plugins, or logs. This is necessary because mounted volumes replace the contents of the original folders. It’s important to note that this may cause issues with customization scripts, such as  initScripts or the use of custom command and arg. Using older versions of Bitnami containers may cause problems because they might lack the necessary bash changes to allow readOnlyRootFilesystem. This setting can be reverted by setting readOnlyRootFilesystem back to false.

Ensuring no pod is without a resource request or limit

Having pods without any resource requests or limits is increasingly discouraged as these containers may deplete node resources. Currently, our charts warn users if the resources object is not set. Additionally, we have the resourcesPreset value to assist users when testing different resource configurations, but this is not recommended for production. With the new changes, we set the resourcesPreset value to the minimum size that operates in our internal testing. This size is not the recommended size by the Bitnami team, but it is the minimum size for basic testing of the solution. It is critical that users set resource values according to your requirements and use case. 


It’s important to note that users already setting the resources value will not be affected by this change. Users who are not setting resources may experience a performance drop in their workloads, as they would be configured to a minimal value for basic testing. In these cases, we strongly recommend users set the resources value before any upgrade. This setting can be reverted by setting resourcesPreset back to none.

Enabling functioning in OpenShift restricted-v2 SCC

Before our changes, Bitnami charts did not function in Openshift restricted-v2 Security Context Constraints(SCC) because they set user and group to 1001. To address this, we enabled the automatic adaptation of the containerSecurityContext and podSecurityContext sections when running in Openshift installations. If the detected platform is Openshift, the values runAsUser, runAsGroup, and fsGroup will be automatically removed, allowing the Openshift platform to select the appropriate user and group IDs.

Users deploying in platforms other than Openshift are not affected by this change. Openshift users who set their runAsUser, runAsGroup, and fsGroup values will find that these are removed during an upgrade. This setting can be reverted by setting global.compatibility.openshift.adaptSecurityContext to false.

For detailed information on all improvements check out this GitHub issue.

Using the Tanzu OSS Health Assessment to understand the impact of these changes

Earlier this month, we launched the Tanzu OSS Health Assessment—a free-to-use tool that provides a comprehensive overview of your open source software dependencies and the potential risks you may face. 

To understand how the new improvements in Bitnami packages can impact your security posture, let’s look at the OSS health assessment report generated before and after the security improvements were implemented.

Let’s look at the OSS health assessment report of a repository with the Helm charts—Nginx, ClickHouse, Grafana and PostgreSQL—before the security improvements were made.


As you can see, the percentage of Misconfigured resources were 54.55%.


Now let’s look at the health assessment report of the same Helm charts after the implementation of our improvement measures.



From the snaps, you can see that misconfigured resources have dropped from 54.55% to 17.39%, implying a significantly better security posture, thanks to the aforementioned improvements.


You can also see that the security vulnerabilities plummeted from 684 to just 4. This drop is due to the fact that the Bitnami Helm charts are sourced from Tanzu Application Catalog with Photon OS as the base image, and not the standard Debian OS (which is the base OS for the community edition of Bitnami packages).


Thus, the recent security improvements in Bitnami-packaged Helm charts combined with Tanzu Application Catalog’s capability to let you use the VMware by Broadcom-maintained Linux distro—Photon OS as the base image can help you improve your security posture and minimize risks from your open source software dependencies.

Next Steps


To solve problems you may have with the Bitnami community packages—including deployment support, operational support, and bug fixes—please open an issue in the Bitnami Helm charts or containers GitHub repository. You can get the latest Bitnami-packaged software from our website, DockerHub (containers and Helm charts), Google Marketplace, Azure Marketplace, or AWS Cloud Marketplace. If you want to use Bitnami packages in production environments for mission-critical use cases, check out Tanzu Application Catalog—an enterprise version of Bitnami Application Catalog with several exclusive features that include base OS customization, app-level customization, Vulnerability Exploitability eXchange (VEX), and more. Take our OSS health assessment today and kickstart your journey toward the optimization of your open source software dependencies.



Monday, March 18, 2024

Bitnami-packaged containers and Helm charts in DockerHub are now signed by Notation

Bitnami-packaged open source software container images and Helm charts available in DockerHub are now signed by Notation, a Cloud Native Computing Foundation (CNCF) incubating project.

In December 2023, we announced that Tanzu Application Catalog, the enterprise edition of Bitnami Application Catalog, started making use of Notation as a tool for signing and verifying the open container initiative (OCI) artifacts (e.g. container images, Helm charts, and metadata bundles) that we deliver. Now, we’re happy to announce the extension of this capability to the community edition of Bitnami-packaged container images in DockerHub as well.

Read on to discover more about Notation, how you stand to benefit from this integration, and how you can verify the Notation signatures in Bitnami packages.

What is Notation?

Docker developed the Docker Content Trust (DCT), a.k.a. Notary V1, in 2015 and subsequently donated it to the CNCF. Notary V1 allows users to sign and verify container images while ensuring the integrity and authenticity of specific image tags through client-side or runtime verification. Notary v1 functions by adding the user's public key to the registry and signing the image with the key's private counterpart before uploading it. Users can then verify the image by comparing the public key against the registry command group's pulled data.

The updated and enhanced Notation seeks to improve upon the shortcomings of DCT or Notary v1. Now users can create and incorporate their own implementations of the specifications into workflows for signing and verifying multiple OCI artifacts (such as software bill of materials, scan results, and container images) using Notation. It’s intended to serve as a cross-registry, cross-industry standard for signing and validating any registry artifact or OCI image. Notation is an implementation of the Notary Project specifications and is a CNCF incubating project

Benefits of signing Bitnami images with Notation

There are several benefits of signing Bignami images with Notation, including the ability to: 


  • Ensure content integrity—By signing our container images with Notation we can guarantee the integrity of the software we deliver. The signatures generated by Notation are based on the content, creating a unique fingerprint for each version of the artifact. Any tampering with the container will result in a failed verification, alerting users to potential security threats.

  • Verify authenticity—Knowing the source of the open source software you use is critical for security and compliance. Notary Project signatures provide a way to verify the authenticity of the artifacts by confirming the identity of the signer. This ensures that your applications are built from trusted sources—Bitnami—reducing the risk of deploying compromised or malicious software.

  • Support interoperability across tools and platforms—Notation plays a vital role in standardizing the representation of signatures. This standardization enables interoperability across different tools and platforms that support the OCI image format without being tied to a specific ecosystem.

Signature verification

To locally verify the signature of a Bitnami-packaged container image, follow the steps below:

  1. Download the “rootCA.cert” file from https://app-catalog.vmware.com/.well-known/notationCA.crt.

  2. Download and install the Notation CLI for your platform from the official releases at https://github.com/notaryproject/notation/releases

  3. Add the Tanzu Application Catalog Root CA certificate:

$ ./notation cert add --type ca --store VMware notationCA.crt

  1. Import the trust policy:

$ ./notation policy import trustpolicy.json

This is an example of the trustpolicy.json file:

$ cat trustpolicy.json

{

 "version": "1.0",

 "trustPolicies": [

    {

     "name": "Tanzu Application Catalog",

      "registryScopes": [ "*" ],

      "signatureVerification": {

       "level" : "strict"

      },

      "trustStores": [ "ca:VMware" ],

      "trustedIdentities": [

       "*"

      ]

}

]

}


  1. Verify the signature of a container image of the Helm chart and check the latest available version or tag from DockerHub.

$ ./notation verify docker.io/bitnami/wordpress:6.4.3-debian-12-r20

Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:6.4.3-debian-12-r20) because resolved digest may not point to the same signed artifact, as tags are mutable.


Successfully verified signature for docker.io/bitnami/wordpress@sha256:4c93c6a8b06ab87c7d5b54d58684157b32cb69e466b9330e7c6460331ff663aa


  1. Use the digest directly.

$ ./notation verify docker.io/bitnami/wordpress@sha256:4c93c6a8b06ab87c7d5b54d58684157b32cb69e466b9330e7c6460331ff663aa

Successfully verified signature for docker.io/bitnami/wordpress@sha256:4c93c6a8b06ab87c7d5b54d58684157b32cb69e466b9330e7c6460331ff663aa


Check out our enterprise version - Tanzu Application Catalog!

If you’re interested in learning more about the enterprise edition of Bitnami Application Catalog - Tanzu Application Catalog, check out our product webpage, solution brief, and additional resources.

Are you going to be at  KubeCon + CloudNativeCon Europe 2024? If so, you can learn how to reinforce your software supply chain security by joining our session, VEXintating your Container Images: The European Way, on March 21, 2024 (Thursday) at 15:25 - 16:00 CET.