Friday, July 29, 2022

How Bitnami, Let's Encrypt and Lego teams Troubleshooted Bncert Issues in Let’s Encrypt Servers

Engineers from the different teams working together to achieve the solution

Authored by Alejandro G√≥mez, R&D Manager at VMware

On 25th June, the Let’s Encrypt team reported an issue in the Bitnami’s GitHub repository for Virtual Machines describing how Bitnami HTTPS Configuration Tool (bncert tool) - the command-line tool for configuring and automatic renewing of HTTPS certificates for Bitnami stacks - was originating an overload of requests in Let’s Encrypt servers. 

This blog post will introduce briefly what the bncert tool is, what Let’s Encrypt is, and explain in detail the issue that was identified by the Let’s Encrypt team. Additionally, we will cover how three teams - Bitnami by VMware, Let’s Encrypt, and Lego teams - worked together to find a solution for this issue. Finally, you will find instructions on how to update your installations to get the latest and more secure version of the bncert tool in your virtual machines. 

What is the Bitnami HTTPS Configuration Tool (bncert tool)?


The Bitnami HTTPS Configuration Tool is a command line tool for configuring mainly HTTPS certificates on Bitnami stacks, but also for common features such as automatic renewals, redirections (e.g. HTTP to HTTPS), and so on. It is included in every Bitnami stack by default. Its main features include generating new certificates and configuring automatic renewal for certificates by using the Lego project, which is a Let's Encrypt client and ACME library written in Go.

What is Let’s Encrypt?


Let’s Encrypt is a free, automated, and open certificate authority - a project under the nonprofit Internet Security Research Group (ISRG). This body gives people the digital certificates needed to enable HTTPS (SSL/TLS) for websites, for free, in a user-friendly way (you can see more about how Let’s Encrypt Certificate Authority works here).

The Issue: An Overload of More than 170K Requests in Let’s Encrypt Servers


After some investigation, both Let’s Encrypt and Bitnami by VMware teams identified the cause of the issue: the current bncert tool configuration. The bncert tool uses Lego - a Let’s Encrypt client - to generate new certificates and configure automatic renewal for these certificates. When the bncert tool is executed, it adds a scheduled job (via Cron) to the “bitnami” user in order to run Lego on a “0 0 * * *” schedule. This configuration provoked unusual spikes and outages in the Let’s Encrypt services from the lego-cli tool as shown in the following chart shared by Let’s Encrypt engineers.



The Lego community added a feature to add a randomized sleep for each renewal request and also implemented the option to specify a user-agent when performing requests to Let’s Encrypt servers. The first feature allows for reducing the spikes and usage issues, while the second allows Let’s Encrypt to identify problematic configurations at scale.

The Bitnami by VMware team has also released a new version of the bncert tool bundling the new version of Lego. That way, the certificate renewal time is randomized and adds the “bitnami-bncert” user agent, which prevents this issue from happening again in any new Bitnami installations. This patch is included in the latest versions of virtual machines of the Bitnami Application Catalog. 

How to Troubleshoot Bncert Tool Issues in Let’s Encrypt Servers


  • To check if you are using the proper version of the tool, execute the following command:

    $ sudo ./bncert-tool --version
    Bitnami HTTPS Configuration Tool 0.7.4 --- Built on 2022-06-08 14:02:48 IB: 21.6.0-202106241241

  • If the version is lower than 0.8.0, then you must update it by running the command below:

    $ sudo ./bncert-tool
    An updated version is available. Would you like to download it? You would need to run it manually later. [Y/n]: y

  • The tool will exit now. To run the updated version run the following command:

     /opt/bitnami/bncert-tool

  • Double-check that the tool was updated to the latest version: 
 
    $ sudo ./bncert-tool  --version
    Bitnami HTTPS Configuration Tool 0.8.0 --- Built on 2022-06-30 15:20:55 IB: 21.6.0-202106241241

In case you generated HTTPS certificates using bncert before this issue, we strongly recommend executing the tool again to renew the existing certificates. This will update the scheduled job for renewing certificates at a random time, as well as setting up the new user agent.

Conclusion


We would like to highlight the great work done by the Lego community for adding those features and thank the Let’s Encrypt team for helping us improve our solutions. The ability of the Lego community and the Let’s Encrypt team to quickly release a new version that added new features to address this issue - and that too, in less than 24 hours - is evidence of the power of the open-source community. They have done an excellent job and we want to recognize them for their great work in creating a highly popular service to enable HTTPS (SSL/TLS) for websites for free.



Wednesday, July 13, 2022

A New Source of Truth for Bitnami Containers

Over the past few months, the Bitnami content team has been working on unifying the source of truth for the different assets that Bitnami provides to its users.  

Currently, users can find the source code for the Bitnami Helm charts in a single repository on Github (https://github.com/bitnami/charts/) while the source code for the Bitnami containers is spread across different repositories for each solution (for example, https://github.com/bitnami/bitnami-docker-wordpress). 

Now, we have adopted a “monorepo approach” to remove the differences in the user experience between Helm charts and containers on GitHub. Moreover, unifying the repositories where their code lives also allows us to unify all the processes involved in the life cycle of their source code. 

Embracing the Monorepo Approach 

In order to unify both repositories, we have opted to extend the monorepo approach to Bitnami containers as we already did with Helm charts. Thus, container users will be able to navigate through the code of all Bitnami containers by checking only a single repository: https://github.com/bitnami/containers/

In addition to an easier way to find the code, users will benefit from:  

  • Having all issues located under the same repository makes it easier to find common use cases and avoids duplication which ultimately improves the support workflow 
  • Having a single and updated place for general announcements, changes in the contributing guidelines, and so on 
  • The possibility of using VMware Image Builder (VIB), a service that allows developers and contributors to verify their Pull Requests (PRs) and thus provide an early feedback loop 

In the coming weeks, the Bitnami team will transfer all valid open issues to the new repository. Then, all the bitnami/bitnami-docker-* repositories will be analyzed in order to identify the valuable issues and migrate them to their new bitnami/containers repo. 

For existing open PRs, we will engage with contributors to find a better way to move their contributions forward into the new bitnami/containers repository. 

In addition, the internal test and release pipeline will be adapted to pull/push changes from the new repository. This is expected to take some time. 

We expect to have a large portion of the data migrated by July 15th.  From that point onwards, repositories will be migrated on a gradual basis. 

How Will Containers Code Look After this Change? 

The existing bitnami/bitnami-docker-* repositories will not disappear. These repositories will be archived in read-only mode so you can continue checking the whole history of the repository changes and existing issues. 

Currently, we plan to delete these repositories after one year, but we would be open to community feedback here. 

Will the source code or the container images change? 

No, the code will be migrated as it is from the different repositories to the monorepo. For example, the source code for PostgreSQL at bitnami/bitnami-docker-postgresql will be moved under the containers/postgresql directory in the bitnami/containers repository. 

Container images will be displayed as usual in the Bitnami DockerHub organization. There won't be any changes to the image itself or to the registries where the image is available. 

Support and Resources  

Looking to learn more or have more questions? Check out the new Bitnami GitHub repository for containers and if you need to get a resolution on issues or want to send us your feedback, please open an issue. A markdown template is provided by default to open new issues, with certain information requested to help us prioritize and respond as soon as possible. 

Also, if you want to contribute to the project, feel free to send us a pull request and the team will check it and guide you in the process for a successful merge.   

Boost your knowledge about Bitnami containers and Helm charts by checking our latest tutorials

Friday, June 10, 2022

We Have Moved! Bitnami Support for Installers, VMs, and Cloud Images is Now Available on GitHub

When you have a problem with a Bitnami solution – whether it’s related to downloading an installer or running a virtual machine or cloud image - you navigate to the Bitnami Community site to find the answers and resolutions you’re looking for, both from Bitnami’s support team and from the community of Bitnami users. Through these interactions, we have built a large community of developers that help each other in solving common issues and thereby help you effectively use our catalog.  

A few years ago, we expanded our offerings to containers and Helm charts and began providing support for them in their natural environment: GitHub. Thus, Bitnami users may have had to navigate to two different sites to solve their issues, depending on the format they were running.  

To provide a unified experience, we will be moving the support for our installers, virtual machines, and cloud images to a GitHub repository as well: https://github.com/bitnami/vms This change will be effective from June 30th, 2022. This will allow the Bitnami support team to meet the needs of our users from just one location: GitHub. 

The Bitnami Community site will remain open in read-only mode until July 15th, 2022.  We will migrate the most relevant content of this site to the new GitHub repository. If you will miss something, please do not hesitate to open a suggestion issue. Your feedback will help us to make this transition smoother!  

How Can I Submit a Request to the New Bitnami Support Page? 

If you are not familiar with GitHub, no worries! Our team has compiled everything you need to know on the README file, and incorporated a pinned issue to guide you on how to work with this repository. 

The following instructions walk you through the process of opening a new issue to receive support from the Bitnami team:  

  • Navigate to https://github.com/bitnami/vms/issues and, in the search box, enter the matter of your issue. Thus, you can check if there is an open issue that matches the topic you are searching for.  

  • GitHub shows only open issues by default. Hence, if you can’t find your topic in the resulting search, try to filter by closed issues as shown below: 

  • If any of the previous searches do not provide any results, then open a new issue. You can choose amongst three different types of issues depending on the problem you are experiencing: How to (if you want to ask us about how to use a Bitnami solution in a specific scenario), Suggestion (if you have feedback that you think will help us to improve our solutions), or Technical issue (if you are facing a problem or detected a bug when using our applications). 

Our support team will be happy to help you on GitHub!