Thursday, September 13, 2018

Meet the Team: Mattia Buccarella

The Bitnami team is a diverse group of talented people distributed all over the world. Get to know them better through this series of blog posts.

Mattia is an Engineer in the WebDev team. Currently,  he is working remotely from Dublin (Ireland), but is planning to move to southern Europe soon.

A Brief Bio

It’s well known that Italy looks like a boot. I come from the very south-east part of the peninsula from the region of Apulia, and I was there until the end of high school.

Ever since I was a kid, I’ve always been interested in computer technologies. At the age of 12, I was already telling people that I wanted to study Computer Science, so my passion was easily visible to anyone who knew me.

At the age of 19, I moved to Pisa (beautiful Tuscany) where I got my degree in Computer Science, at the University of Pisa, followed by a Master in Computer Science and Networking (MCSN) at the prestigious Scuola Superiore Sant’Anna. After 1 year of working as a researcher at the University of Pisa, I moved to Ireland to work at Amazon Data Services. After 3 long years, I finally and fortunately landed in Bitnami.

Why did you join Bitnami and what excites you about working here? 

Let’s start with the fact that I always worked in the cloud ecosystem. For some unexplainable reason, I always knew that the “Cloud” would  be “the future”. When I was thinking about leaving Amazon, I started to research companies that would fit my interests best. After I found Bitnami, my search was quickly over. How could any other company compete with a place that allows me to work on the technologies that I love the most? I understand that not everyone has the ability to make their passion their day job, so I feel very lucky to now to have a job that coincides with the same work that I would want to do during my free time. This is what excites me the most about Bitnami.

In addition to that, the flexibility of a remote job fits my long-term plan to move to a different country since the grey skies in Dublin and Ireland have been starting to affect my southern italian self.
When I joined, my colleagues played a significant role in convincing me to move to Spain. Their overwhelming support in combination with my willingness to move to another country has made me decided that I’d like to join my team in the Bitnami Seville office and work as a local there! I look forward to being in the office with the team soon.

What are you working on?

Getting up to speed with all the technologies that the webdev team is currently working on is not an easy task. It has been unbelievable to learn how such a small number of people have been carrying out all of these projects at such a high speed! As a WebDev Engineer, I am working on the technology that publishes our cloud applications into the most popular cloud platforms such as AWS, Azure, Google, and Oracle. Scalability is the primary keyword for this project.

Since the company is growing, I have the opportunity to do more than software development as well. I have the ability use the knowledge that I learned at large scale companies, like Amazon, to help Bitnami build for the future. With the chance to help mold the processes of Bitnami, I am continuously driven to review my ideas to ensure that I do it better, bigger, and make it so for myself and for the future of the company.

What do you like to do for fun?

I am a gym person. At the same time, I am the definition of laziness. Due to these contradictions, when I have spare time, I either work out at the gym or sit on my chair to play some video games. It depends on my mood or the weather :P

If I am not at the gym or playing video games, I like to play guitar. In spite of the excitement that this brought to my new teammates, I feel like I am at a very basic with this instrument. Before the next company all-hands, I’ll need to practice in order to satisfy my team’s expectations since I’ve heard that Bitnami likes to do jam sessions during the company parties!

Interested in working with Bitnami and Mattia? Apply for one of our open positions!

Wednesday, September 12, 2018

Deprecation of AWS Paravirtual Images

Nine years ago Bitnami started providing cloud images for open source software. The first cloud supported by Bitnami was AWS.

A lot has changed since then, including the virtualisation technology used to provide AMIs. AWS started using a technology called paravirtual (PV) images, but later moved to a  new virtualization type, hardware virtual machine (HVM).

Today, HVM is the recommended solution for virtualization due to its superior performance. Also, after recent security vulnerabilities in the Linux Kernel, AWS has published a security bulletin recommending users to migrate to HVM instances due to insufficient system protection to address process-to-process risks in PV images.

For these reasons and because we are committed to provide optimized and secure images, Bitnami has decided to stop providing updates for PV images after October 1st. Existing images will continue to be available, however they will no longer be updated with security fixes.

Affected Servers


Deprecating PV images does not impact running servers, however we strongly recommend you  move your applications to an instance based on a HVM image.

In order to check if you are running a paravirtual server, you can look for the “Virtualization” property in the EC2 console description tab for your instance. In the official AWS documentation you can find the recommended upgrade path when moving from PV to HVM machines.

Affected Accounts and Next Steps


New regions and current instance types only support HVM. Please note that some of instance types can only be launched in a VPC.

If your EC2 account was created before 2013-12-04, you might need to configure your account to support VPC. If that is the case, we advise that you:

  • Create a new AWS account. This will create an EC2-VPC account which comes with a default VPC in each region.
  • Select a region that you haven’t used before to launch your servers. When using a new region a VPC will be created by default.
  • Contact AWS support to migrate your EC2-Classic account to EC2-VPC. This may require removing old resources in your account.

You can find more information about enabling VPC support in your account and how to migrate a linux instance from EC2-Classic to a VPC in the official AWS documentation.
Questions

For questions relating to your account, we recommend to contact AWS directly.Bitnami does not have access to your EC2 account details.

For any questions about PV support, please don’t hesitate to contact us in our community forum.

Tuesday, September 11, 2018

Integrating Stacksmith with Atlassian Bamboo, Bitbucket and Jira

By Wojciech Kocjan, Solution Artichect at Bitnami

Bitnami Stacksmith is a tool that automates and optimizes the packaging of your applications for deployment to cloud and container platforms. It also continuously monitors your applications for updates and patches, and maintains them so they remain up to date and secure.

At Bitnami, we understand that enterprise companies rarely conduct release management manually - it is a complex process used to coordinate developers, components, and steps that include application development, code check in, compiling and testing, versioning and releasing, and deploying. Typically, this process is automated using one or more Continuous Integration (CI) or Continuous Deployment (CD) tools to streamline the process.

Stacksmith performs a few critical pieces of the release management process. In certain use cases, like when trying to repackage your traditional applications so you can replatform them from your data center to the cloud, using Stacksmith as a ‘stand-alone’ solution can be the right approach. With that being said, Stacksmith provides significant value in use cases that involve ongoing application development, deployment and maintenance, which is why we felt it was  critical that Stacksmith be able to fit easily into any existing CI / CD or orchestration tooling you already utilize.

This article goes through the process of creating a fully automated setup where Bitnami Stacksmith is part of the continuous integration (CI) cycle and generates up-to-date cloud deployable assets. Here, I’ll use a Java web application that uses a database, but the process is applicable for different setups and use of the various stack templates that Stacksmith provides.

My setup will consist of the following components:


The following image shows how the process of application development, compilation and packaging will look like:


Why introduce Stacksmith to the CI / CD pipeline?


Atlassian Bamboo is a powerful continuous integration server that lets you automate release management in your CI / CD pipeline. It provides the basic tooling to define projects and how they’re built, but it’s up to its users to define the exact stages and tasks that should be run.

In many cases, when companies set up Bamboo, they focus on the development part of the lifecycle - compiling the application, then running unit tests to ensure it is working properly. Bamboo comes with multiple tasks for compiling different types of applications, running tests and understanding its results. In these instances, the continuous deployment part is given less focus - in many cases, it simply involves copying the built artifacts to a staging and/or production server, and restarting it.

Bitnami Stacksmith can improve the CD part of the pipeline by enabling packaging of the application and creating deployable assets for various public clouds as well as Kubernetes. It creates outputs specific to the target platform - such as AMIs and CloudFormation templates for AWS, VM images and ARM templates for Azure, and Docker images and helm charts for Kubernetes. Stacksmith outputs are also optimized for each platform - leveraging load balancers, databases and other services available to those specific public clouds.

Stacksmith also lets you include your IT and security policies in the process - including image OS and configuration hardening, network policies, agents and any other policies, and applies them as it builds the deployable assets. This ensures that every image is built according to Security’s specification. In addition, Stacksmith can analyze whether those have changed, and either notify you or automatically repackage your application with the latest policies.

Stacksmith packages the application and provides environment-specific deployment mechanisms that launch the application along with all of its dependencies - setting up networking, databases, load balancers and other resources. The deployments can be used to automatically or manually launch the application. It can also be used to manage existing deployments - providing a smooth way to upgrade as well as downgrade the application in development, QA or production environments.

By having assets that allow new deployments of the application to be easily launched, it’s possible to perform full deployment of the application in the cloud, run automated tests of the application and its entire environment for each build, and validate the exact setup it will be run in.

The following illustrates how performing continuous packaging fits into the CI / CD pipeline:


Bitnami Stacksmith provides additional value by constantly monitoring the components that went into the deployable assets. It monitors trusted sites for new releases and security updates for your application's components, provides alerts when available, and offers a simple repackaging process.

Integrating Jira, Bitbucket, Bamboo and Stacksmith


First, I’ll set up Atlassian Jira, Bitbucket, Bamboo and Bitnami Stacksmith and show how the suite of tools work together to provide a CI / CD pipeline that includes the creation of deployable assets. Next I will show how this pipeline performs end to end testing by launching the application in the cloud and verifying that it is working correctly. Finally I’ll update the deployment provided the application or deployable asset tests were successful.

The components and setup


The first piece of software is Atlassian Jira. It’s an issue and project tracker that is used to track projects - specifically to track the progress of application development as well as all the issues related to it.

Second application is Atlassian Bitbucket - it’s used for storing the source code of the application. It’s configured to be linked to the Bitbucket and Jira servers, following the process in Atlassian documentation.

Third element is Atlassian Bamboo - it is the continuous integration (CI) server that allows you to define build plans and run them to create cloud deployable assets from the application’s source code. In order to have all the data available across all Atlassian apps, Bamboo is linked with Jira as well as Bitbucket. And since Stacksmith will be used to do the actual packaging and creation of the deployable assets, Bamboo will also be linked to Stacksmith.

The final and centrally important element is Bitnami Stacksmith - the tool that will actually create the assets that will be deployed to the cloud. All that is needed to set up Stacksmith is to sign in (if you don’t already have a subscription you can sign up for a free trial), and go through a very easy process of onboarding your AWS or Azure account.

Defining the CI pipeline - from source code to deployable assets


Now that all of the applications are set up, I can configure them so that the process of going from source code of the application to deployable assets that I can run in the cloud is fully automated.

Building application from source code with Atlassian Bamboo


I’ll start with Atlassian Bamboo. I’ve created a very basic build plan that packages the application - simply checking out the source code from Bitbucket server and using Maven to build the application from source code.




Bamboo will now monitor the source code in the Atlassian Bitbucket server, and will perform a new build every time the application source code changes. Thanks to the links that have been set up in this integration, all of this data is also accessible from Jira.

Creating deployable assets using Bitnami Stacksmith


Next I need to set up Stacksmith, so it can be used to build the deployable assets. The first step is to set up the application definition in Stacksmith - this involves selecting the specific stack template you will use. The stack template informs Stacksmith about the type of application you are packaging and its requirements. In this case, I will select the  "Java Tomcat Application with DB (MySQL)" stack template, which informs Stacksmith to handle Tomcat set up and create the database, leveraging cloud-native resources if possible (such as RDS on AWS or Azure Database for MySQL on Azure). This step only has to be done once, and can be done via Stacksmith Web UI, its APIs, or the Stacksmith CLI.

The next step is to integrate Atlassian Bamboo with Stacksmith. This can be done with the stacksmith-cli tool that provides a convenient way to trigger rebuilds of an application in Stacksmith, retrieve the build log in real time, and get results back. All of the Stacksmith APIs, as well as the CLI tool, are available to all Stacksmith users - including those using the free trial.

The stacksmith-cli is used inside a CI tool such as Atlassian Bamboo. To configure integration with Stacksmith, a simple YAML file with configuration for the CLI tool is created - Stackerfile.yml. Here is a sample contents of the file:

appId: bitnami/apps/7cbf4630-8c6d-0136-d7fb-66a676282717
appVersion: 1.0.0
files:
  userUploads:
   - target/myapp-1.0.0.war
  userScripts:
    boot: boot.sh

The appId specifies the application identifier that’s provided by Stacksmith. The appVersion field specifies the version to pass to Stacksmith. The files section specifies the files to upload - the application file(s) as well as the scripts that Stacksmith should use.

The target/myapp-1.0.0.war file was built by Maven (as described earlier in this article) in Bamboo. The boot.sh file is a boot script and includes application-specific logic and specifies how to provide the application, with details required for connecting to database.

The Stackerfile.yml can either be written manually or generated programmatically - such as using the Maven filtering functionality to create the file with appropriate placeholders.

The last thing that is needed is to create a helper script called build-with-stacksmith.sh that calls the *stacksmith-cli* to build the application. Here’s the contents of the script:

#!/bin/bash

set -eu -o pipefail

stacksmith auth login --access-token "${STACKSMITH_ACCESS_TOKEN}"
stacksmith build

The Stackerfile.yml, boot.sh and build-with-stacksmith.sh files are committed to the repository with the source code of the application, which makes it possible to call them directly from Bamboo.

Setting up Bamboo to trigger a build in Stacksmith is easy. It requires setting up the stacksmith-cli binary on the Bamboo server. Then all that is needed is to create an additional step in the build plan - running the build-with-stacksmith.sh script from the source code repository.




The STACKSMITH_ACCESS_TOKEN environment variable is passed to the script and contains an API token that can be retrieved using the stacksmith-cli.

This new step can be tested by manually running a new build, which calls the CLI tool and  triggers a new build in Stacksmith. The link and live information about the build is also shown in Bamboo at build time, and is accessible after the build finishes.



Bamboo is already monitoring the source code repository in Bitbucket for changes - any new change to in the application code will now trigger a rebuild, performing a full automation from source code to deployable assets.

Continuous deployment to the cloud


At this point our application is packaged and Stacksmith has created the charts or templates that will be used to launch the new deployment or update an existing deployment - Helm charts for Kubernetes, CloudFormation templates for AWS and ARM templates for Azure. This enables you to create a test deployment and run additional tests against the application as well as upgrade the application version in the development, QA or production environments. The Stacksmith API or the stacksmith-cli tool can be used to retrieve the outputs from the packaging process.

Having an simple way to deploy the application makes it easier to run tests of the entire cloud deployment. All that’s needed is a relatively small script that can be run from Atlassian Bamboo - to simply retrieve the template URL, launch a test deployment and run tests against the application.

Below is a sample shell script that runs tests using AWS. Deploying to Azure or using helm charts is very similar, except they use different client tools.

#!/bin/bash

# unique name for the deployment
stack_name=teststack-$(date +%Y%m%d%H%M%S)-$$

# template URL is retrieved from Stacksmith and passed as argument
template_url=$1

# launch the template in AWS or fail immediately
aws cloudformation create-stack --stack-name "${stack_name}" \
  --template-url "${template_url}" || exit 1

# (wait for cloudformation stack to be launched)

# retrieve URL of the application from AWS
url=$(aws cloudformation describe-stacks --stack-name "${stack_name}" | jq -r \
  '.Stacks[0].Outputs[] | select(.OutputKey == ("PublicDnsName")) | .OutputValue')

# run test and store whether it succeeded
./application-tests.sh "${url}"
code=$?

# delete the deployment
aws cloudformation delete-stack --stack-name "${stack_name}"

exit ${code}

The application-tests.sh script should perform application specific tests, getting the URL that the application is available. It should indicate if the tests have failed using non-zero exit code. The tests can include anything - such as interacting with the application via APIs and/or web automation tests.

Similar to the build-with-stacksmith.sh file, this script can be committed to the application source code repository and set up to be run as a task in Bamboo.

Deploying an updated version of the application in various environments is very similar - the only thing that is needed is to update the existing deployment - using aws cloudformation update-stack command instead of create-stack. For example:

#!/bin/bash

set -eu -o pipefail

# unique name for the deployment
stack_name=teststack-$(date +%Y%m%d%H%M%S)-$$

# template URL is retrieved from Stacksmith and passed as argument
template_url=$1

# launch the template in AWS or fail immediately
aws cloudformation update-stack --stack-name "deployment-staging" \
  --template-url "${template_url}"

# (wait for cloudformation stack to be updated)

The script above will update the existing deployment in AWS and wait to ensure it has succeeded. It should also be committed to the application source code repository and invoked as a task from Bamboo. It should be invoked after the tests of the deployment have passed.

The packaged application can also be manually launched from the Stacksmith Web UI. Launching the application into any cloud simply requires choosing the right build and clicking the Launch button:


This starts the process of deploying the packaged application, redirecting the user to the cloud provider’s console and finishing the process - for AWS this is done using the AWS CloudFormation Console. For Azure, this starts the process of deploying an ARM template in the Azure Portal.


Monitoring application for security issues and updates


Stacksmith provides functionality to ease the maintenance of your applications. It continuously monitors the components that make up the deployable assets (such as system packages), checking for updates and known security vulnerabilities. It alerts the user about security issues as well as whether updates are available that resolve the security issues. It also provides convenient manual options or fully-automated and continuous ways to repackage the application.

This ensures a smooth process for upgrading your application, making it easy to repackage a fixed version of the application and update existing deployments to fix any security issues.

Summary


As this article has demonstrated, it is easy to utilize our set of APIs and CLIs to integrate Bitnami Stacksmith with Jira, Bitbucket and Bamboo. This combination enables you to set up a fully automated pipeline that starts with source code and ends with the creation of assets you can deploy to the cloud.

The Stacksmith APIs and CLIs can also be used to integrate with other popular CI / CD tools. It is easy to apply the same steps we discussed here to an environment that utilizes Jenkins, GitLab and many other solutions.

Package, deploy, and maintain your own application today. Sign up for the Stacksmith 30 day free trial and find out how easy it is!

Get started with Stacksmith now!

Have questions about how Stacksmith can fit with the CI / CD tools you currently use?  Or want a personalized demo?  Contact us at enterprise@bitnami.com.