Introducing Pipelines: Mission Control for DevOps

Nov 10, 2015

What are Pipelines used for and why are several companies building them—including us?

Let’s begin with a brief story. If you’re a software engineer, it’s easy to imagine yourself as the protagonist of this story. It begins with a rush of action: You write some code in your favorite text editor. You setup version control with with a solution like Git or Mercurial. You push your repository to a service like Bitbucket or GitHub.

You elect to use a SaaS service for building and deploying your code, and you hear about Distelli and decide to give it a go. After signing up, Distelli has you create an application. During the creation process, you’re asked to link specific branches of your source control repository to the application and create a distelli-manifest.yml file in your repository.

This essential branch linkage, coupled with the Distelli manifest, turns the application into a vessel for your code—making it possible to build, test, and deploy the application with Distelli. If your application builds and the tests pass, the build artifacts are packaged into what’s called a release—which can be deployed to any server that has the Distelli agent installed.

Enter environments. Environments encapsulate a set of servers and environment variables. You may choose to create three environments for your application: beta, gamma, and production. You could deploy the latest release of your application to the beta environment, find that everything is in order, and promote the release to the gamma environment for a final round of quality control before promoting to production.

The process repeats itself—again and again—and the process is invariably good: It ensures that bugs don’t ship to production, code quality remains high, and that every action is auditable.

But there’s a lot going on. There’s numerous interfaces to contend with: applications, environments, releases, builds, deployments…etc. There’s a whole lot of clicking, parsing, and clicking some more to fabricate a mental model of the entire process. What release does this environment have? Who promoted the active release in the gamma environment to production? Did that build fail or succeed? Software delivery—from source control to server—is a complicated process.

Mission Control

An interface that attempts to display this process is what we—and the industry—have come to call a Pipeline. It was a great challenge to design and develop a Pipeline interface that includes every aspect of the software delivery process outlined above—and offer more than just cursory details.

A Pipeline interface must deliver valuable information in real-time and give customers an intuitive way to operate off such information with swift precision. As I was designing and developing Pipelines, I began to refer to it as “Mission Control” for DevOps:

Pipelines

How it works

A Distelli Pipeline reserves the left column for applications. Every other column is called a stage, which contains environments. You can create pipes between applications and environments, and between environments themselves, which can be used for promoting releases to environments in different stages.

Pipelines Pipelines

Every aplication or environment in a stage gives you the most critical details at a glance. To drill down further into such details, a dialog window swoops in from the right side of the screen to provide more context and interactions. Stages are by no means immutable after they’re created. It’s possible to drag environments between stages in order to re-structure your Pipeline on the fly:

Pipelines

Pipelines

The use of color is very important: blue, yellow, red, and green communicate the state of builds and deployments. Because a Distelli Pipeline synchronizes with our backend services, the colors will subtly change when updates are received. No refreshing necessary.

Pipelines

Every part of the Pipeline interface helps you build, test, and deploy your code with confidence. To learn more and get started with your own Pipeline, feel free to read the amazing Knowledge Base article written by my coworker Brian: Introduction to Pipelines. I hope you enjoy using Distelli Pipelines. Please feel free to reach out if you have any questions.