Keeping Your Application Secrets Safe

Jan 20, 2016

Distelli uses layers of encryption to keep your applications’ secrets secure, and lets you decide which servers can decrypt these secrets. We sat down with Brian Maher, our Principal Software Engineer, to talk about the challenges of creating a solution that puts your security first.

There’s one main reason that the current wave of DevOps adoption is taking hold: speed. Every problem that the DevOps industry aims to solve is in aid of this speed:

  • Continuous integration solutions let you swiftly making new code available for testing.
  • Automated testing solutions let you get the most basic tests out of the way, so that software quickly moves through the testing cycle.
  • Continuous delivery solutions let you quickly go from “all tests passed” to “released to customers”.

And so on. For the purpose of keeping speedy deliveries going, then, you need to make sure that even as you’re eliminating human intervention in your software lifecycle, your data stays protected.

Keeping Secrets

Continuous integration or automated testing notwithstanding, your application likely needs access to a variety of secrets—a database password here, an API token there, and so on. The recommended way to store these secrets is on your server—in an environment variable, for example—so that you can change these secrets easily without having to rebuild and redistribute your application.

But how do you get a secret to your server? Ordinarily, this is part of *server provisioning*—setting up a server to host your application. When you install the programs your application needs and configure the server, you’d also add these secrets to the environment variables.

Now, let’s say you want to deliver your application at a faster pace—automate whatever you can, and reduce overhead wherever possible. For example, instead of running your own servers, you start using Amazon EC2. With EC2, you can automatically add new instances to respond to increased load, and to turn them off when they’re not being used. This is excellent; but now you have a problem: you still need to manually add the secrets to your environments, and your customers aren’t going to wait while you get stuff set up.

Ideally, you want any new EC2 instance to easily access these secrets when it starts. One logical (and instantly regrettable) solution is to put a secret file with in your source control repository. It’s logical, because it’ll be put into your release package when it’s built, and can be deployed to your server with the package. Instantly regrettable, because putting sensitive information into your repository makes it available to anyone who’s tracking your project, and essentially makes your secret not secret at all.

Another possible solution is to host the secret on a server. When you build your release package, you add a command to download the secret and put it in the release package. While this is less bad than putting the secret in your repository, you now have to worry about two new things:

  • The security of the server that has your secrets. If an attacker gains access to the server or its traffic, your secret is compromised.
  • The security of the release package. The build is only the first step—the release package needs to be transmitted to your destination servers, and if an attacker intercepts it, your secret is compromised.

So much for eliminating overhead.

The Problem: Adding Secrets to the DevOps Automation Workflow

Our problem statement, then, was simple: your servers need to have easy access to your secrets, and we need to keep those secrets safe, all while minimizing manual intervention. The goal:

  • Provide a secure way to store and transfer secrets—certificates, credentials, and so on—to your servers. You shouldn’t have to invest valuable resources to create your own secure server, nor should you have to trust a third-party storage service to keep your data secure.
  • Make sure that your secrets aren’t compromised if an attacker targets Distelli or any of the services Distelli uses. For example, we store your release packages on an Amazon S3 bucket. While we expect that Amazon has the resources and the desire to keep their service secure,the ‘widely trusted’ appeal of S3 also makes it a pretty high-value target for attackers. Your secrets need to be safe if one of those attackers gets lucky.

In keeping with our mission to let you focus on your customers’ problems, we’ve built a solution that lets you secure your secrets with minimal overhead. If you’re ready to start securely delivering secrets to your servers, see our knowledge base article, Using Secure Packages to Deploy Secrets.

What follows here is the story of how we built security into your build-deploy-release workflow.

Solving the Problem: Encryption Everywhere

What do you do to make sure that your data stays secure while it’s flying around in the cloud, outside your control? Three things:

  • Encrypt data locally, and only then store it on the cloud.
  • Transmit the encrypted data using secure protocols like TLS or HTTPS.
  • Decrypt the data only when it needs to be read, and only on authorized machines.

Which is simple enough when you’re manually uploading and downloading data, but needs special consideration when you throw automation into the mix.

Let’s start with a typical example: you want to store an important document—your tax return, perhaps—on an online storage service like Dropbox or Google Drive. Both are widely used and trusted, so you have some assurance that your document is secure. But there are still risks that have nothing to do with the services themselves: if someone gets access to your account—by attacking your password, or even purely by accident—your document is at risk. Time for another layer of security.

The most straightforward way to protect this document is to use the first principle we talked about earlier: encrypt the data locally before uploading it. To do this, you use a symmetric key to encrypt your file before uploading it, and decrypt it when you need to refer to it. Even if someone has your password, without the symmetric key, all they get to see is gibberish.

But now, this gives an attacker a single target to compromise everything: lose the key, all is lost. Someone steals your computer? No more encryption key, no more data. Want to share the file with someone? You need a secure channel to share the encryption key too.

This is a critical problem for building and deploying your applications: we needed to make sure that (a) your release package is encrypted, and (b) once encrypted, only your servers can decrypt the package—meaning that we need a secure way to get the encryption key from Distelli’s servers to yours.

Protecting the Encryption Key

The next question is, “How do we secure the symmetric key?” The industry standard is to secure the key using *asymmetric encryption*—so that we can encrypt the symmetric key with a public key, and only authorized entities with the private key should be able to decrypt that key.

Authorized entities, meaning servers that you control.

Which leads us to the next layer:

  • Encrypt your release package, using a release key.
  • Create a key pair for the destination server: a public key to encrypt, and a private key to decrypt.
  • Encrypt the release key using the public key, and send the encrypted key and release package to the server.
  • The server uses the private key to decrypt the release key, and uses the release key to decrypt and deploy the package.

Voila. And in essence, this is how Distelli secures your release artifacts. The rest is how we adapted this approach for scale, performance, and general best practices.

Scaling and Good Practices

The above solution is straightforward enough if you’re building just one application and deploying it to one server. As you add applications and servers, it gets messy.

For example, if you have 2 destination servers, we’d have to store:

  • Two encrypted copies of every release key—one for each server.
  • Two public keys—one for each server.

Every release gets its own key, so if you build your application 10 times a day, we have 10 release keys and two encrypted versions of each per day—making that 20 new release keys a day.

By the 10th day, Distelli is tracking 200 encrypted release keys. A good practice is to change your encryption key periodically; so if you created new encryption keys on the 10th day, all 200 release keys need to be re-encrypted with the new public keys. If you have 20 destination servers, that number is 2000, and so on.

This is not undoable, but it’s certainly not efficient.

In addition, if the release key needs to be encrypted with the public key for your destination server, all your destination servers must be up and running at the time of the build—which isn’t a fair assumption if you’re, say, starting new EC2 instances to respond to increased loads.

There’s a more efficient solution:

  1. We consolidated key management to a few servers, which you set up in your infrastructure. The key management servers—instead of your destination servers—are used to encrypt the symmetric keys. This is more scalable than having an encrypted copy of the release key for every single destination server, and doesn’t depend on having all your destination servers up at the time of the build. This approach also lets you protect these servers with more controls than any other server.

  2. We added the account master key, which is a single key for your account, and use it to encrypt the release key. This lets us enable key rotation with a much lower key management overhead: The release key is encrypted with the master key, and we store one encrypted version of the master key for every key manager—so now, if you have 10 releases and 2 key management servers, instead of re-encrypting 20 release keys, we just have to re-encrypt master key twice, and use that master key to re-encrypt the 10 release keys. Much more efficient.

And there we are.

Stepping Back

This is what we learned, above all else: key management gets complicated. We started with a standard best practice, but then came the many permutations of servers, key managers, keys encrypted with other keys, and so on.

And because we took on those complications, we’ve ended up with a better product:

  • You can now use a secure package to deliver secrets to your servers, without saving or transmitting them in plaintext.
  • We’ve checked off the data protection requirements of the Sarbanes-Oxley act: data at rest is always encrypted, and only encrypted data is transmitted.
  • Your release packages remain secure if any access controls fail, either through malicious or accidental action. To have a chance at seeing your builds, an attacker would have to compromise Amazon S3, your key management servers, and reverse-engineer Distelli Agent. This considerably increases the cost of compromising your releases.
  • Setting up secure applications is straightforward, and adds little (if any) overhead to your Distelli setup.

The best part: you don’t have to rely on any new third-party services—only servers you own can decrypt your data.

More Resources