Securing Ansible SSH Keys

Managing the SSH keys Ansible uses to connect to remote machines can be challenging. Placing keys on the Ansible Controller makes those keys difficult to rotate. A machine with the ability to connect to all network machines is a high value target. Let’s look at a better way to manage SSH keys: move those keys into a secure vault. Retrieve keys only when Ansible needs a particular key.

Setting the Stage

Everyone’s environment is going to look a bit different. Let’s start by defining some context for our example environment:

We have two applications: Foo and Bar, and two different environments: staging and production. The servers in a given application environment share a common SSH key pair. As such, we have four key pairs: staging Foo, staging Bar, production Foo, and production Bar.

Getting Started

The easiest way to get started with Conjur is to follow our Get Started tutorial.

Describing our Policy

Let’s create a simple policy to allow our Ansible Controller to retrieve the four private SSH keys. We also want our admin to be able to update the private keys.

# ansible.yml

- !policy
  id: ansible
  body:

  # define a YAML collection `keys` to hold our ssh key variables
  - &keys
    # create variables to hold the private key
    - !variable staging-foo-ssh_private_key
    - !variable staging-bar-ssh_private_key
    - !variable production-foo-ssh_private_key
    - !variable production-bar-ssh_private_key

    # create a group with permission to retrieve SSH keys
    - !group secrets-users

    # Give the `secrets-users` group read/execute privilege (read provides visibility, execute allows retrieval of the value) to the variables stored in the `keys` collection defined above
    - !permit
      role: !group secrets-users
      privileges: [ read, execute ]
      resource: *keys

    # A layer defines a group of one or more machines. We'll use this group to give our Ansible Controller access to the above SSH private keys
    - !layer

    # Define a host factory for this layer. A host factory allows us to generate a short lived, IP restricted token to auto-enroll our Ansible Controller into our Ansible layer
    - !host-factory
      layer: [ !layer ]

    # Now let's give this layer the ability to retrieve the SSH private key
    - !grant
      member: !layer
      role: !group secrets-users

The above is a bare-bones policy to get us started. We can refactor this policy later to give us more flexibility and control.

Add SSH private keys

Now that we’ve created variables, let’s add our private SSH key into Conjur.

# log into Conjur
$ conjur variable values add ansible/staging-foo-ssh_private_key "$(cat ssh_keys/foo/staging_rsa)"
$ conjur variable values add ansible/production-foo-ssh_private_key "$(cat ssh_keys/foo/production_rsa)"
$ conjur variable values add ansible/staging-bar-ssh_private_key "$(cat ssh_keys/bar/staging_rsa)"
$ conjur variable values add ansible/production-bar-ssh_private_key "$(cat ssh_keys/bar/production_rsa)"

Give the Ansible Controller a Conjur Identity

Now we have a policy that defines our SSH keys and gives our Ansible Controller permission to retrieve those keys. Next we’ll need to give this server an identity, and enroll that server into the correct group to ensure it can retrieve our private keys. We’ll do this using Ansible.

First, install the Conjur Role:

$ ansible-galaxy install cyberark.conjur-host-identity

Next, update the Ansible Controller playbook to include our Conjur role:

# playbooks/ansible_controller.yml
- hosts: ansible
  roles:
    - role: configure-conjur-identity
      conjur_appliance_url: 'https://conjur.myorg.com/api'
      conjur_account: 'myorg'
      conjur_host_factory_token: "{{lookup('env', 'HFTOKEN')}}"
      conjur_host_name: "{{inventory_hostname}}"
  ...
  

You’ll notice the environment variable HFTOKEN. We’ll generate a Host Factory token for our Ansible layer and populate HFTOKEN prior to running this playbook. The Host Factory token will auto-enroll our Ansible Controller into the ansible layer.

A Host Factory Token is a short-lived key (optionally IP-restricted), used to auto-enroll a host (server) into a layer. Host Factory tokens allow automated systems to enroll new instances when they are provisioned without requiring human intervention.

In this example, we’ll generate a token valid for 3 minutes and restricted to an IP subnet (10.0.1.0/24).

$ hf_token=$(conjur hostfactory tokens create --duration-minutes 3 --cidr 10.0.1.0/24 ansible/ansible  | jq -r '.[0].token')

With our Host Factory token generated, we have three minutes to enroll our Ansible Controller. Let’s give it an identity!

$ HFTOKEN="$hf_token" ansible-playbook "playbooks/ansible_controller.yml"

Our conjur Role does a couple of things:

1. Connects to Conjur, and using the generated Host Factory Token, creates a Conjur host and enrolls that host into the ansible layer (which has permission to retrieve the remote SSH keys).
2. Creates two files: /etc/conjur.conf (contains information about the location of Conjur), and /etc/conjur.identity (contains authentication information needed to retrieve secrets from Conjur).
3. Installs Summon, and the Summon-Conjur provider, which makes retrieving and providing secrets to a process a breeze.

Run Playbooks

Once the cyber playbook has been run successfully, we’re ready to update Ansible to use the keys stored in Conjur to connect to our remote hosts:

$ summon --yaml 'SSH_KEY: !var:file ansible/staging/foo/ssh_private_key' bash -c 'ansible-playbook --private-key $SSH_KEY playbook/applications/foo.yml'

What’s happening here? Let’s run through the steps:

1) Summon connects to Conjur, using the /etc/conjur.conf and /etc/conjur.indentity files for authentication.

2) As we’ve given our host execute permission for the Conjur variable ansible/staging/foo/ssh_private_key, Summon retrieves its value and creates a temporary file with the variable’s contents. The temporary file’s path name is stored in the SSH_KEY variable.

3) The temporary file is passed to the ansible-playbook process through the argument --private-key.

4) Once the ansible-playbook process completes, the temporary file is removed from the system, leaving no trace of our SSH key.

Optionally Summon can be used with a secrets.yml file. For our Ansible example, our secrets.yml file might look like:

production-foo:
  SSH_KEY: !var:file ansible/production-foo-ssh_private_key  

production-bar:
  SSH_KEY: !var:file ansible/production-bar-ssh_private_key

staging-foo:
  SSH_KEY: !var:file ansible/staging-foo-ssh_private_key  

staging-bar:
  SSH_KEY: !var:file ansible/staging-bar-ssh_private_key

With the above secrets.yml file on our Ansible Controller, we can run our playbook as follows:

$ summon -e staging-foo bash -c 'ansible-playbook -i $SSH_KEY playbook/applications/foo.yml'

Running in Production

We’ve illustrated this example using the Open Source, hosted version of Conjur. When using Conjur for managing credentials in production, we strongly recommend you run Conjur with SSL (check out the TLS Tutorial), or check out Conjur Enterprise, which includes TLS, high availability, LDAP integration, and a number of other features aimed at larger organization.

In Summary

Managing SSH keys for Ansible doesn’t have to slow your team’s velocity. Move those keys into a secure store like Conjur, and access them with Summon. Using credentials on-demand brings security and flexibility without destroying your organization’s existing workflows.

Questions? Connect with us on the CyberArk Commons.