Tuesday, October 4, 2022

Analyzing the Bitnami Container Catalog Size: Findings and Next Steps

Authored by Alejandro Gómez, R&D Manager at VMware


At Bitnami, we are always thinking of new ways to improve our catalog and provide the best experience to our users. An example of this is the creation of a new source of truth for Bitnami containers, which you can read about in this blog post. These are some of the actions we on the Bitnami engineering team have taken to enhance the current size of the Bitnami container catalog:
  • Analyze distroless and docker-slim as options to decrease containers’ base image size
  • Getting insights from Dive and considering the use of multi-stage builds to increase the performance of the image
  • Test docker-build –squash as an option to reduce the container image size
In this post, we share our findings as well as the option the team finally adopted to boost the performance of our containers by reducing the image size.

Distroless and docker-slim

With the aim of improving the catalog, our team ran an analysis to determine the best option for reducing the size of container images nowadays. We considered either moving into distroless as the base image for our containers or using docker-slim as a tool to shrink the container images.

We found it very difficult to embrace the distroless approach since some applications, like Apache, required more dependencies to be compiled, which would add more complexity to the catalog.

On the other hand, the analysis of docker-slim wasn’t good for some assets, like PostgreSQL 14, because it didn’t produce good results in the tests—as a result of lack of configuration from our side—as shown below:

docker-slim build --compose-file docker-compose.yml --target-compose-svc postgresql --include-path /opt/bitnami --http-probe-off --include-shell --include-bin /bin/sh --include-bin /usr/lib/x86_64-linux-gnu/libedit.so.2 --include-bin /usr/bin/touch  --include-bin /bin/mktemp --exclude-pattern "/bitnami/postgresql/**" --exclude-pattern "/opt/bitnami/postgresql/tmp/**" --exclude-pattern "/tmp/**" --pull  --path-perms "/opt/bitnami/postgresql/tmp:775#0#0"

We got a container image reduction of 33.4%:

bitnami/postgresql.slim   14       0ce6542042ff   1 second ago   181MB
bitnami/postgresql        14        0336c8e4fba4   23 hours ago   272MB
The container worked without issue with the default helm install values. But it failed in the architecture=replication configuration because some binaries were missing. Based on the usage of docker-slim, we couldn’t calculate it properly because we were using a basic configuration without tuning it. Even though we have seen significant reductions in the applied cases—some of them worked perfectly, like Pytorch or Apache—we also have found that they failed in the detection phase in most cases, which requires a substantial amount of case-by-case manual configuration. This would mean spending a lot of work tuning all our containers to make sure they are properly configured to pass all these tests.

Dive and multi-stage builds

To test how to shrink the size of a Docker image, we have used dive, a tool for exploring a Docker image, layer contents, and discovering ways to shrink the size of your Docker/OCI image by calculating the efficiency ratio of a Docker image. This is calculated by checking the number of files that have been repeated between layers (for example, by doing chmod operations). For this reason, we decided to run this analysis on the heaviest assets we have in the catalog, such as Pytorch. In the case of Pytorch, we achieved an efficiency of 51.13% because of a chmod operation we are running in a different layer that duplicates one of the layers.

The Bitnami engineering team also analyzed the usage of multi-stage builds to check whether the steps above could help. First, we ran a basic test using the WordPress container and its whole Dockerfile as reference and converted both in a wordpress-builder step. Then, we built the final Dockerfile by installing only the dependencies and copying from the wordpress-builder the result of the /opt/bitnami folder, where all the binaries are installed. The result of that test wasn’t as good as expected because, when running the WordPress container, an error occurred:

wordpress_1  | cp: cannot create regular file '/bitnami/wordpress/wp-config.php': Permission denied

COPY --from" does not preserve the permissions in our test:

Although there was a previous work at Multi-stage COPY --from should preserve ownership/permissions, that option was not working as expected and extra execution steps were needed to, again, adjust the file permissions, which could lead to the same problem we had before running the tool.

You might wonder, “Why not optimize the Dockerfile directly?” The answer is not easy. At Bitnami, we don’t create the Dockerfile files manually; we have a system through which each asset is modeled and generates the files automatically. This is why we shouldn’t tune each Dockerfile asset, because we must think about the future maintenance and ensure that if an update of the asset comes, it will work.

docker build --squash analysis

The last option we evaluated was to add the --squash flag when running the docker build phase. This would squash newly built layers into a single new layer, preserving the base image layer. Thus, we shouldn’t change the container nor remove files. It would only squash all the layers into a single one, which sounds reasonable, but we had to make sure if that made sense or not.

For this analysis, we built the whole catalog with and without the --squash option, then we checked the container image sizes again. This was the output data we got:
  • 312 container images built (remember some products have different versions like, for example, MariaDB)
  • 36 container images (11.54%) of BAC had a size reduction of at least 20%:
    • 3 of them, more than 50%
    • 9 of them, between 40% and 50%
    • 6 of them, between 30% and 40%
    • 18 of them, between 20% and 30%
  • 135 container images (43.27%) had a size reduction minor than 20% but major than 1%
  • The rest (141 container images, 45.19%) had a reduction minor than 1% (most of them are built from scratch, so it makes sense not to see any benefit)
For the entire Bitnami Application Catalog, we got a benefit of 25,219 MB (25,219 GB saved) in raw mode; remember that when we push the container images into a registry, they are usually compressed. This first output meant we were going to save an important amount of traffic and storage size per container release.

Bitnami builds their containers using VMware Image Builder, so we needed to check whether the vulnerability scan would keep working. We did this by using some non-squashed and squashed container images. The Aqua Trivy scanner worked as expected (VMware Image Builder uses both Trivy and Anchore Grype as the vulnerability scans), throwing the same output in the tests.

After confirming that the container images worked as expected, we checked the repository storage impact too by testing whether or not performing all steps mentioned above was worthwhile for end users. To confirm that point, we did a test pushing the most reduced container images in both flavors (squashed and non-squashed). After pushing squashed and non-squashed container images, we confirmed a storage reduction of more than 40% (when an image is pushed, it’s compressed first, but we also observed benefits after compressing the container image).

On the other hand, we also discovered that squashing container images was worthwhile, and that also improves the speed when an end user pulls more than one Bitnami Docker image. Based on several tests and use cases, the download of the squashed images was 32% faster than the regular ones. This can be explained because the base image is the only shared layer due to the cadence. Although the download of layers can be done in parallel, docker doesn’t extract images in parallel.

Conclusions and next steps

After doing the analysis and seeing the pros and cons of each option, preserving layers is not so critical for the main use case of Bitnami users. The possibility to reuse layers is really low and also the base OS that comes in the base image—that is cached—has the new OS changes. We have decided to implement the docker --squash option in our catalog to reduce the container’s images size and to help our end users, reducing the time you have to wait to get the container image ready to be used and alleviating the repositories storage in case someone wants to use our images as a base image on their own containers.

We are aware there could be other use cases where preserving the container layers could be required or beneficial, but based on our data, the approach that we have adopted would be beneficial in the majority of cases.

This will not be our only action to improve the catalog. We will keep working hard to improve it, and we will share the numbers of the usage of this implementation with the community.

Support and resources

Looking to learn more or have more questions? Check out the new Bitnami GitHub repository for containers, and if you need to get a resolution on issues or want to send us your feedback, please open an issue. A markdown template is provided by default to open new issues, with certain information requested to help us prioritize and respond as soon as possible.

And if you want to contribute to the project, feel free to send us a pull request, and the team will check it and guide you in the process for a successful merge.

Boost your knowledge about Bitnami containers and Helm charts by checking our latest tutorials.

Friday, September 23, 2022

Kubeapps Extends Package Repository Management

New Package Repository API and UI in version 2.5.0

Authored by Pepe Baena, R&D Manager at VMware

The Kubeapps team is continuously looking for ways to improve the user experience when browsing and deploying applications on Kubernetes clusters. The new Kubeapps release ﹣ version 2.5.0 ﹣ extends the repository management feature by including Carvel packages. Now, users will be able to configure package repositories for both Helm and Carvel packages in just a few clicks. 
Keep reading to learn more about the key points that make it possible: a new Package Repository API and a revamped form that allows you to easily configure package repositories directly from the UI. 

Context of this Enhancement

Before releasing version 2.4.3, Kubeapps was mainly focused on Helm chart management, and as a  consequence, was tied closely to Helm repositories and OCI registries for Helm chart storage. When we added support for Carvel packages and Helm charts via Flux, it became necessary to standardize the management of different package repository types in Kubeapps. This brought the following challenges for the team:

  • We needed to design and implement a new API, similar to the Kubeapps core API for managing packages, but in this case for managing package repositories as well as ensuring that the new API satisfies the requirements of the existing Helm functionality.
  • We needed to implement a Package Repository API for the different plugins (Helm, Flux, and Carvel).
  • We needed to revamp the current UI form to standardize the experience when configuring existing repositories (Helm and OCI) as well as Carvel and Flux sources.

Let’s see in detail what implementation strategy and steps were performed to successfully add this new feature to Kubeapps.

Design and Implementation of a New Package Repository API

The first step we followed was to design a new Package Repository API that allowed us to manage new package repositories on Kubeapps. The main component of the domain model is the PackageRepository (a repository of Kubernetes application packages that are available for installation). A repository may be served by any one of the backends supported by the configured plugins (Helm, Flux, and Carvel plugins), including, but not limited to:

  • a Helm chart repository (e.g. https://charts.bitnami.com/bitnami ) – via Helm or via Flux plugins
  • an OCI helm repository ﹣ via Helm plugin or via Flux plugin
  • a Git repository containing artifacts ﹣ gzip compressed TAR archives, via Flux plugin
  • object storage buckets containing artifacts coming from S3 compatible storage such as Minio, Amazon S3, Google Cloud Storage, Alibaba Cloud OSS, and others ﹣ via Flux plugin
  • a Git repository or HTTP tarball of installable packages ﹣ for Carvel kapp﹣controller plugin
  • a Docker image reference for an Image or ImagePackageBundle ﹣ used by Carvel kapp﹣controller to define a repository of installable packages

PackageRepositories may be either global (cluster-wide) or namespace-scoped, in which case only users with access to that namespace will have access to the repositories﹣Flux does not have the concept of global repositories. Furthermore, PackageRepository may be public (i.e. require no authentication to access its contents) or private (i.e. require some form of authentication to access its packages). Taking all those requirements into consideration, the Repositories API definition was defined and made publicly available in order to provide a complete reference to the new API. In addition, the API is also available for each Kubeapps instance and includes a documentation portal accessible using user credentials: https://<kubeapps url>/#/docs.

Once the initial version of the Package Repository API was defined, the next step was to implement the API for each available plugin in Kubeapps ﹣Helm, Flux, and Carvel plugins. We decided to begin the work by implementing the API for the Flux plugin ﹣as any other could have been selected ﹣ while continuing in parallel the development of the API for Helm and Carvel. Fortunately, we could speed up the development thanks to the pluggable architecture of Kubeapps implemented in prior versions. During the implementation for each plugin, we verified that the new API satisfied the needs of the UI and found some gaps in the existing form that needed to be addressed.

Revamping of the Kubeapps UI

When the new Package Repository API had been totally implemented, it was time to adapt the UI to comply with the new API, and at the same time, to simplify the user experience in managing package repositories in Kubeapps. 

The main challenge we found was how to group and display all the fields in a way that the Kubeapps UI will be able to guide users through the configuring process. The revamping process of the form was tackled in three stages:

  1. Using the new repository API in the existing UI to manage new API calls, report issues back, and improve the API when needed. The main goal of all these actions was to ensure feature parity between the new API and what Kubeapps supports with a consistent implementation across plugins.
  2. Completing support for secret management in private repos. To do so, we had to unblock all the issues related to authentication and the existing inconsistencies across plugins that were identified in previous stages.
  3. Working on code and UI enhancements. In this last stage, the team focused on cleaning the code including the removal of pending calls to deprecate kubeops, adding some improvements in the form such as the scope field to select for global or namespaced repos, and fixing some issues identified when using the new UI in different scenarios.

As a result, the new UI for Package Repository management includes the following sections:

Basic information

In this section, users will include information about the repository such as litname, URL,  packaging format, and storage type. 

To improve the user experience, we have added a new field in this form: the repository scope. This was previously set by default depending on the namespace. Adding an extra field to set the scope of the repository allows Kubeapps users to perform this configuration explicitly.

Note that the storage types field for Helm differs from the one set for Carvel packages as mentioned above. Refer to the next image to see how it looks when you  select Carvel packages as the packaging format: 


Once the basic information has been entered in the form, depending on whether the repository where the packages live is public or private, Kubeapps will require authentication data. We originally planned to support scenarios where the user may choose to provide a reference to an existing secret or string cert_authority, which Kubeapps would convert into a new secret on the backend but we found that approach to be complex, particularly in those cases that require an update in the Package Repository. To simplify the process, the form now includes only two options for secrets: 

a. The user will manage secrets
b. Secrets will be managed by Kubeapps

These options are set on a per-repository basis; thus, different users can use whatever option they prefer in the same Kubeapps installation.  When the Auth and/or TlsConfig options are configured, the user has the choice to select an existing secret (option A) or to enter the credentials (option B). Any updates thereafter must be according to the original option selected.

  • If the secret was created by choosing option A, the plugin will return the secret name and any update must be done via secrets; 
  • If the secret was created by choosing option B, the plugin will return the credentials and any update must be done by providing the credentials again.

The plugin will automatically detect and manage whether the package repository is using option A or B. In the case of option B, it is recommended to leverage the Kubernetes ownerReference mechanism along with a managed-by annotation.

From a UI perspective, the revamped form includes a section for Authentication with all these different options:


Finally,  some advanced fields have been included to configure different aspects of the repository such as the synchronization interval ﹣ available for Flux and Carvel plugins ﹣ CA certificate, skip TLS verification, or passing credentials to 3rd party's URL as shown in the image below:

Conclusion and Next steps

Thanks to the implementation of the Package Repository API and the pluggable architecture in Kubeapps, the team was able to move forward and boost development when the API was defined. The new form lays down the foundation for making Kubeapps simple by guiding users through the configuration process.

Next, we plan to tackle challenges such as the support for OCI registries in the Flux plugin, adding sync intervals for the Helm plugin as well as exploring repository configuration for multi-cluster and air-gapped environments. 

I would like to highlight the great work done by the Kubeapps team in implementing the new Package Repository API and revamping the user interface.  

Support and Resources

Since Kubeapps is an open-source software project, support will be provided on a best-effort basis. To get a resolution on issues such as deployment support, operational support, and bug fixes, please open an issue in the Kubeapps GitHub repository. A Markdown template is provided by default to open new issues, with certain information requested to help us prioritize and respond as soon as possible. Also, if you want to contribute to the project, feel free to send us a pull request, and the team will guide you in the process of a successful merge.

In addition, you can reach out to Kubeapps developers at #kubeapps on Kubernetes Slack (click here to sign up).

For more information about the topics discussed in this blog post, refer to the following links:

Wednesday, September 14, 2022

Fix Available: Load Problems in Let’s Encrypt Servers due to Bitnami Cloud Images

Our colleagues from the Let’s Encrypt team have informed us that they have identified an issue with the certificate renewal process that is causing load problems in Let’s Encrypt servers.  

This issue affects Bitnami users that are using the Bitnami HTTPS Configuration tool (Bncert tool) to configure HTTPS on their Bitnami cloud deployments. To solve this issue, you must update your Lego installation as explained below. 

How to update your Lego installation 

A fix has been included in the Lego tool version 4.8.0 which adds a random 0-8 minute delay to avoid such spikes in the specific 0:00 minute. However, Bitnami cannot propagate this change to users unless they execute the tool. 

In order to avoid problems in case the renewal fails for several days in a row, and to avoid load problems, users should follow the steps below:  

  • Execute Bncert again to renew the certificates. The tool will request to be updated - press “Yes”. This will also randomize the times in the crontab and add the user-agent to the crontab. 

 $ curl -L https://github.com/go-acme/lego/releases/download/v4.8.0/lego_v4.8.0_linux_amd64.tar.gz | tar xz -C /opt/bitnami/letsencrypt l 


  • Manually update the lego version by running the following command:  

$ curl -L https://github.com/go-acme/lego/releases/download/v4.8.0/lego_v4.8.0_linux_amd64.tar.gz | tar xz -C /opt/bitnami/letsencrypt l 


  • Randomize the renewal time. E.g. from 0:00: 

0 0 * * * sudo /opt/bitnami/letsencrypt/lego ... 

         To a random time, such as 21:40: 

40 21 * * * sudo /opt/bitnami/letsencrypt/lego … 

Once executed, the command {{sudo crontab -u bitnami -l}} should show something like this: 

 40 21 * * * sudo /opt/bitnami/letsencrypt/lego ... 

Where 21:40 is the new randomized time - you will probably see a different value - at which point the renewal will happen every day. 

Support and Resources   

Looking to learn more or have any questions? Check out the new Bitnami GitHub repository for virtual machines. If you need to get a resolution on issues or want to send us your feedback, please open an issue. A markdown template is provided by default to open new issues, with certain information requested to help us prioritize and respond as soon as possible.  

To learn more about how to generate and install a Let’s Encrypt certificate for a Bitnami application, refer to this tutorial.