Engineers from the different teams working together to achieve the solution
Authored by Alejandro Gómez, R&D Manager at VMware
On 25th June, the Let’s Encrypt team reported an issue in the Bitnami’s GitHub repository for Virtual Machines describing how Bitnami HTTPS Configuration Tool (bncert tool) - the command-line tool for configuring and automatic renewing of HTTPS certificates for Bitnami stacks - was originating an overload of requests in Let’s Encrypt servers.
This blog post will introduce briefly what the bncert tool is, what Let’s Encrypt is, and explain in detail the issue that was identified by the Let’s Encrypt team. Additionally, we will cover how three teams - Bitnami by VMware, Let’s Encrypt, and Lego teams - worked together to find a solution for this issue. Finally, you will find instructions on how to update your installations to get the latest and more secure version of the bncert tool in your virtual machines.
What is the Bitnami HTTPS Configuration Tool (bncert tool)?
The Bitnami HTTPS Configuration Tool is a command line tool for configuring mainly HTTPS certificates on Bitnami stacks, but also for common features such as automatic renewals, redirections (e.g. HTTP to HTTPS), and so on. It is included in every Bitnami stack by default. Its main features include generating new certificates and configuring automatic renewal for certificates by using the Lego project, which is a Let's Encrypt client and ACME library written in Go.
What is Let’s Encrypt?
Let’s Encrypt is a free, automated, and open certificate authority - a project under the nonprofit Internet Security Research Group (ISRG). This body gives people the digital certificates needed to enable HTTPS (SSL/TLS) for websites, for free, in a user-friendly way (you can see more about how Let’s Encrypt Certificate Authority works here).
The Issue: An Overload of More than 170K Requests in Let’s Encrypt Servers
After some investigation, both Let’s Encrypt and Bitnami by VMware teams identified the cause of the issue: the current bncert tool configuration. The bncert tool uses Lego - a Let’s Encrypt client - to generate new certificates and configure automatic renewal for these certificates. When the bncert tool is executed, it adds a scheduled job (via Cron) to the “bitnami” user in order to run Lego on a “0 0 * * *” schedule. This configuration provoked unusual spikes and outages in the Let’s Encrypt services from the lego-cli tool as shown in the following chart shared by Let’s Encrypt engineers.
The Lego community added a feature to add a randomized sleep for each renewal request and also implemented the option to specify a user-agent when performing requests to Let’s Encrypt servers. The first feature allows for reducing the spikes and usage issues, while the second allows Let’s Encrypt to identify problematic configurations at scale.
The Bitnami by VMware team has also released a new version of the bncert tool bundling the new version of Lego. That way, the certificate renewal time is randomized and adds the “bitnami-bncert” user agent, which prevents this issue from happening again in any new Bitnami installations. This patch is included in the latest versions of virtual machines of the Bitnami Application Catalog.
How to Troubleshoot Bncert Tool Issues in Let’s Encrypt Servers
- To check if you are using the proper version of the tool, execute the following command:
$ sudo ./bncert-tool --version
Bitnami HTTPS Configuration Tool 0.7.4 --- Built on 2022-06-08 14:02:48 IB: 21.6.0-202106241241
- If the version is lower than 0.8.0, then you must update it by running the command below:
$ sudo ./bncert-tool
An updated version is available. Would you like to download it? You would need to run it manually later. [Y/n]: y
- The tool will exit now. To run the updated version run the following command:
/opt/bitnami/bncert-tool
- Double-check that the tool was updated to the latest version:
$ sudo ./bncert-tool --version
Bitnami HTTPS Configuration Tool 0.8.0 --- Built on 2022-06-30 15:20:55 IB: 21.6.0-202106241241
In case you generated HTTPS certificates using bncert before this issue, we strongly recommend executing the tool again to renew the existing certificates. This will update the scheduled job for renewing certificates at a random time, as well as setting up the new user agent.
Conclusion
We would like to highlight the great work done by the Lego community for adding those features and thank the Let’s Encrypt team for helping us improve our solutions. The ability of the Lego community and the Let’s Encrypt team to quickly release a new version that added new features to address this issue - and that too, in less than 24 hours - is evidence of the power of the open-source community. They have done an excellent job and we want to recognize them for their great work in creating a highly popular service to enable HTTPS (SSL/TLS) for websites for free.