Bitnami Apache Airflow Multi-Tier Now Available in Azure Marketplace

Originally published on the Azure blog on April 9th, 2019.

A few months ago, we released a blog post that provided guidance on how to deploy Apache Airflow on Azure. The template provided a good quick start solution for anyone looking to quickly run and deploy Apache Airflow on Azure in sequential executor mode for testing and proof of concept study. However, the template was not designed for enterprise production deployments and required expert knowledge of Azure app services and container deployments to run it in Celery Executor mode. This is where we partnered with Bitnami to help simplify production grade deployments of Airflow on Azure for customers.

We are excited to announce that the Bitnami Apache Airflow Multi-Tier solution and the Apache Airflow Container are now available for customers in the Azure Marketplace. To see how easy it is to launch and start using them, check out the quick video tutorial below:

We are proud to say that the main committers to the Apache Airflow project have also tested this application to ensure that it was performed to the standards that they would expect.

Apache Airflow PMC Member and Core Committer Kaxil Naik said, “I am excited to see that Bitnami provided a Airflow Multi-Tier in the Azure Marketplace. Bitnami has removed the complexity of deploying the application for data scientists and data engineers, so they can focus on building the actual workflows or DAGs instead. Now, data scientists can create a cluster for themselves within about 20 minutes. They no longer need to wait for DevOps or a data engineer to provision one for them.”

What is Apache Airflow?

Apache Airflow is a popular open source workflow management tool used in orchestrating ETL pipelines, machine learning workflows, and many other creative use cases. It provides a scalable, distributed architecture that makes it simple to author, track and monitor workflows.

Users of Airflow create Directed Acyclic Graph (DAG) files to define the processes and tasks that must be executed, in what order, and their relationships and dependencies. DAG files are synchronized across nodes and the user will then leverage the UI or automation to schedule, execute and monitor their workflow.

Introduction to Apache Airflow Architecture

Bitnami Apache Airflow has a multi-tier distributed architecture that uses Celery Executor, which is recommended by Apache Airflow for production environments.

It is comprised of several synchronized nodes:

● Web server (UI)
● Scheduler
● Workers

It includes two managed Azure services:

● Azure Database for PostgreSQL
● Azure Cache for Redis

All nodes have a shared volume to synchronize DAG files.

DAG files are stored in a directory of the node. This directory is an external volume mounted in the same location in all nodes (both workers, scheduler, and web server). Since it is a shared volume, the files are automatically synchronized between servers. Add, modify or delete DAG files from this shared volume and the entire Airflow system will be updated.

You can also use DAGs from a GitHub repository. By using Git, you won’t have to access any of the Airflow nodes and you can just push the changes through the Git repository instead.

To automatically synchronize DAG files with Airflow, please refer to Bitnami’s documentation.

Bitnami’s Secret Sauce - Packaging for Production Use

Bitnami specializes in packaging multi-tier applications to work right out of the box leveraging the managed Azure services like Azure Database for PostgreSQL.

When packaging the Apache Airflow Multi-Tier solution, Bitnami added a few optimizations to ensure that it would work for production needs.

● Pre-packaged to leverage the most popular deployment strategies. For example, using PostgreSQL as the relational metadata store and the Celery executor.
● Role-based access control is enabled by default to secure access to the UI.
● The cache and the metadata store are Azure-native PaaS services that leverage the additional benefits those services offer, such as data redundancy and retention/recovery options as well as allowing Airflow to scale out to large jobs.
● All communication between Airflow nodes and the PostgreSQL database service is secured using SSL.

To learn more, join Azure, Apache Airflow, and Bitnami for a webinar on Wednesday, May 1st at 11:00 am PST - Register Now.

Get Started with Apache Airflow Multi-Tier Certified by Bitnami Today!

Supporting Enterprise Architectures with Azure Database Services

Authored by Michael Murphy, Product Marketing Manager

Bitnami supports Microsoft enterprise customers in multi-tier environments in numerous ways, and actively works with the Azure Database Services team to provide enterprise solutions and promote best practices across our portfolio of products and projects. In light of the recent announcement regarding the availability of two new Bitnami packaged solutions pre-configured with Azure Database for MariaDB, I wanted to write this accompanying blog post to provide a bit more detail. Not only about the new offerings, but also about how Bitnami supports Azure Database Services across our product and project portfolio. Here’s how we make it easy for you to add Azure Database Services to your applications and projects.

Open source applications in the Azure Marketplace

As a long time Microsoft partner, Bitnami provides a wide range of pre-packaged, easy to deploy open-source applications and development stacks to the Azure Marketplace. ‘Bitnami Certified’ has become synonymous with ‘trusted, secure, and easy to deploy’. Many of these are multi-tier solutions.

What we just announced extends this support for enterprise architectures. We worked with the Azure Database Services team to further simplify the deployment of two of the most popular and widely deployed applications, WordPress and Drupal. While both of these applications have long been packaged and available from Bitnami in the Azure Marketplace, these new packages offer the applications pre-configured as a multi-tier solution that include Azure Database for MariaDB support.

These packages take the complexity out of configuring what would typically be a sophisticated configuration setup. Deploying one of these applications is now as easy as scrolling through the Azure Marketplace, selecting the offering, and clicking ‘launch’.

Launching one of these application packages lets you leverage the stability and security of Bitnami’s production grade application package while harnessing the power of a fully managed Azure Database, allowing you to scale quickly and reach global distribution without worrying about costly downtime.

Your applications with Stacksmith on Azure and AKS

Stacksmith, from Bitnami, is a product for packaging your own multi-tier applications. Stacksmith lets you enforce best practices during packaging, including your configuration requirements for the use of Azure Database Services, including CosmosDB and Azure Database for MariaDB.

Stacksmith provides reference architectures for Java Tomcat, .NET Core and other Linux applications that include Azure Database Service configuration with an ARM Template out-of-the-box. These architectures are customizable to your requirements, or you can define your own best practices and deployment policies. Stacksmith then codifies them into the packaging process, ensuring that your best practices for using Azure Database Services, including their settings for geo-redundancy, retention policies, tagging and networking configuration are included. Once defined, Stacksmith can apply and maintain these policies and best practices over time across your application portfolio, applying them every time the application is packaged or updated.

Doing so simplifies the process via automation, and enables you to implement and enforce database best practices for your applications. And since Stacksmith includes multi-format support, you can utilize a single packaging process to create artifacts and their database configurations for deployment to Azure and / or AKS / Kubernetes.

Check out the video below for a quick demo: 

Containers / Kubernetes applications on AKS

There are numerous ways Bitnami helps enterprise customers connect database services on AKS. As I mentioned above - Stacksmith can be used to package your applications for multi-tier deployments to your Kubernetes clusters.

You can also use Kubeapps with Open Service Broker for Azure and Stacksmith, in the following way. Kubeapps is an open source project with a web-based graphical user interface that helps you to discover Kubernetes applications and deploy them to your Kubernetes cluster on AKS. Microsoft has partnered with Bitnami to integrate Kubeapps with Open Service Broker for Azure, to allow you to create and present Helm charts that leverage Azure-native PaaS services as part of the architecture. For example, you can deploy your application tier on AKS and an Azure Database Services backend, getting the best of both worlds - a highly scalable application with a highly available database. To maintain your application with this database configuration to ensure it stays up to date and secure over time, you can use Stacksmith. You can read more about Open Service Broker for Azure here.

As you can see, we have been busy supporting our enterprise customers and continue to expand the scope of this support to make it ever easier to set-up and deploy complex environments that leverage first party Azure services.

To learn more, register for the “Create an Application-Centric DevOps Experience with Stacksmith and Azure OSS Database Services” webinar with Andrea Lam, Product Manager of Azure Database for MySQL/MariaDB at Microsoft, on January 29th at 9:00 am PST.

Helm Summit In Review

Authored by Rick Spencer, VP of Engineering

The Helm project was initiated initially started, roughly, by Deis, Google, and Bitnami. Deis and Google combining packaging technology in the first iteration of the tooling, and Bitnami providing the actual packages and the expertise in packaging. It then evolved that Google devs focused more on providing the testing, and Deis devs primarily drove the development of the tools, with Bitnami devs contributing charts, and related tech, such as the linter.

Microsoft acquired Deis last year. I was involved in a (very) few of the discussion regarding Helm 3 and Helm summit, and it seemed clear to me that Microsoft was providing resources and people to plan and execute the summit. I have to commend Microsoft for their very light touch at the summit. Other companies might have used the summit as an opportunity to push their brand on the community, but this summit was very community focused. It was all about contributors.

You could really see this on Day 2 when our own Miguel (an Emeritus Maintainer of Helm) moderated a panel of the core devs. As you would expect many, but not all, of the core devs were from Deis and currently working for Microsoft. And very active contributors such as Matt Farina (also a core dev) don’t even work at any of the three initial companies that started Helm. Even the conference MC was not a Microsoftee, but rather from Nike, who are apparently heavy users of Helm. I think that this effort to keep the project a real community open source project will be a strong contribution to the success and longevity of the project.

Day 1 was very “backward facing,” dedicated to reviewing the history of the project, but also with a strong focus on presentations from users discussing how they use Helm. It was fun to hear so many Bitnami sponsored projects (monocular, sealed secrets, our charts, etc…) mentioned throughout the day. We see the numbers of users and contributions growing but it was nice to hear how much the community is embracing our contributions.

Day 1 wasn’t just about celebrating Helm, these users brought up problems and challenges. This was important for Day 2, when the conference switched to planning Helm 3.0.

For example, Ubisoft discussed their heavy usage of Helm. They noted that they struggled at times because Helm assumes charts will be deployed “by themselves” so Helm has limited composability. Related, they also said that they find collaborating on deployments is not as well supported by Helm as they would like. For example, one team cannot easily provide a standard implementation of something like networking or logging across all teams. They are looking at ksonnet to see if that can help. They are also looking at monocular to help with visibility.

I also particularly enjoyed the Chart Museum presentation. Chart Museum provides a highly functional chart repo, and the code seems very community friendly.

After lunch on Day 2, the conference switched to working sessions to discuss Helm 3. To set up for this, 2 of the core maintainers set the groundwork by describing requirements for Helm 3.

While I thought the requirements were reasonable, I did feel that they were a little too prescriptive, and limiting. It felt that they were trying very hard to limit the scope of the changes, and therefore limit the project.

However, Brian Grant from Google went on to make a strong case that Helm should be decomposed into a set of unix-like tools rather than one server and one client, and ensure that each of these tools is built in a really “kubernetes way” (my read was that meant using all of the appropriate parts of the current and future kubernetes API).

In my view, this would be a stronger approach to the tooling. I think it would be important to maintain compatibility with existing Helm 2 charts, as that is where the lion’s share of the user effort has gone so far. However, having a set of smaller tools with strictly defined inputs and outputs could keep the community from splintering as different teams adopt different technologies to fit their needs (I’m thinking of things like jsonnet, ksonnet, and kubecfg). Additionally, I think this approach could result in a deeper relationship between the Kubernetes and Helm communities.

I don’t think the Helm project or community is ready to take this leap in version 3.0. It seemed that there was more “low hanging fruit” in improvements that the community is eager to tackle first.

Overall this was a great event. I look forward to the progress we’ll make as a community on Helm 3.0 given the direction that was set and the long term future of Helm as a project with the group that was represented in Portland last week.

Container Trends – Bitnami User Survey 2017 (Part 2)

Survey Says: Kubernetes

Name the top 5 container orchestration solutions.

Top answer on the board: Kubernetes

The rise of Kubernetes as the leading container orchestration tool should come as no surprise. It’s the topic of the day with Amazon and Microsoft recently joining the CNCF. In a few short years, we’ve seen hundreds of companies join the ecosystem building or modifying solutions to support kubernetes (check out the recently updated CNCF Landscape) and we’ve even seen the early days of acquisitions beginning to happen. With all that is being said and written, it’s still good to back up a trend with some good old-fashioned data.

As promised (see Container Trend Part 1), in our second post covering our recent user survey we’re taking a look at container orchestration trends. In our last blog post, we showed the increase in interest, highlighting a more than 2x increase in production container usage from 2016 to 2017. As that increase in container usage was happening, what impact did that have on how containers were being managed? Of course, we’d expect some increase in usage of container orchestration to match that growth.

We asked our users “What Container Orchestration System(s) does your company use?” and the results were surprising in a few ways. First and foremost was the enormous growth of Kubernetes. And while Mesos usage doubled, it still pales in comparison to new entrant offerings like AWS Elastic Container Service and Azure Container Service. Docker Swarm showed significant growth over that period as well, perhaps due to Swarm being included in the Docker 1.12 release. The least surprising bit of data was the sharp decline in users with no container orchestration, which is supportive of the shift from dev/test to production.

Figure 1. Container Orchestration Adoption - 2016 vs 2017

Key Stats:

• 115% growth in businesses using Kubernetes
• 100% growth in Mesos
• AWS Container Service overtaking Docker Swarm in less than 1 year

Digging deeper into container orchestration, we wanted to understand the scenarios in which the various platforms are being used. Knowing there was such a huge shift to production environments in the past year and seeing the impact that had on orchestration adoption above, we wanted to understand if there was preference for one platform over another as users make that move. For the most part, platform selection for dev/test is aligned with production. Focusing specifically on 2017 in this data set, we can see that Kubernetes, AWS and Azure usage all increased a few percentage points over their general adoption numbers when users were focused on production usage, with the largest number of users selecting Kubernetes.

Figure 2 – Container Orchestration Adoption 2017 – Dev/Test vs. Production

Key stats:

• Kubernetes is platform choice for over 50% of existing production container deployments

If you are making a decision on where to invest as you build out a container strategy or you’re looking for tools that can help you manage your Kubernetes environment, you’ve come to the right place. Bitnami can get you started on your journey with pre-packaged container images from our vast catalog of ready-to-run applications and we’re actively developing a contributing to a number of leading edge kubernetes projects.

Stay tuned for more from our 2017 Bitnami user survey. Next time we’ll break down container orchestration a little further and look at mixed usage …we’re just getting started.

Introducing ksonnet, an Open Source configuration experience for Kubernetes

We are pleased to announce ksonnet today, an open source tool for configuring applications running on Kubernetes clusters that we have built in collaboration with our friends from Box, Microsoft and Heptio.

Bitnami's mission is to make awesome software available to everyone. We originally started providing easy to use native installers for popular open source server software. We've quickly expanded into providing virtual machines, cloud images and, more recently, containers.

Kubernetes has emerged as the leader in deploying production container workloads. Though Kubernetes can be thought of as an orchestration system, it has turned into a full-fledged platform that others can build on. A large ecosystem of contributors has emerged, providing tooling around monitoring, security, management and any other aspect of building and maintaining Kubernetes clusters. In particular, Bitnami has been involved with the Helm package manager and related projects such as Monocular and Kubeless, the Kubernetes-native serverless framework.

Internally, we have been early adopters of Kubernetes ourselves. In the process of migrating all of our infrastructure to Kubernetes, we ran into scenarios that pushed the limits of what current solutions could deal with. As a result, we have ended up creating our own tooling to help define and manage complex Kubernetes deployments. Around the same time, Heptio was working on a similar project and approached us to combine efforts, resulting in ksonnet.

ksonnet is an open source configuration tool for configuring applications in Kubernetes based on the jsonnet templating library. It is designed to be easy to use, yet extensible and powerful enough so it can cover as many scenarios as possible.

Our goal is that ksonnet will help lower the barrier of adoption for Kubernetes and will continue to evolve and integrate with the rest of the Kubernetes ecosystem. Though it has just been released, it is already being worked on by an active group of contributors that includes Red Hat, CoreOS, Box and Microsoft. We are particularly excited about the integration with the Helm project, allowing the generation of Helm charts that support ksonnet as an alternative to existing templates.

Heptio and us are excited to share ksonnet with the community, helping push Kubernetes further into the mainstream. Give it a try today and let us know what you think!