Tutorial: Sharing Credentials Across Multi-cloud with Centralized Secrets Management

As organizations of all cuts and sizes are migrating to the cloud, there is a need to risk-assess the outcomes of this process. One of the many risks involved is when they end up throwing all the eggs in one basket and move most of their business operations under a single vendor, creating a vendor lock-in effect.

In order to avoid that, some companies have adopted a loosely coupled system with either mixed or hybrid cloud topologies; a multi-cloud approach where operations are spread across two or more cloud providers.

With that approach, another risk factor comes into question: In a multi-cloud environment, how do we manage credentials such as SSH keys, user credentials and API tokens, in a way that avoids vendor lock-in? The answer: By using a dedicated secret management solution, of course, that has no ties or biases against a particular cloud provider. By having a more customized and platform-agnostic option for handling sensitive data, we can perform tasks like secrets rotation or fine-grained access control and sharing.

This post provides a look at Cyberark Conjur, which is perfectly designed for handling secrets in multi- or hybrid-cloud deployments. In a step-by-step guide that includes how to install a Conjur appliance in a Virtual Machine, we demonstrate how to load policies that control credentials across cloud Accounts (AWS and Azure, specifically), and how to test the retrieval of those credentials in a unified and secure way.

Let’s get started!

Step 1 – Install Conjur

The first thing we need to do is deploy a Conjur Server in a VM that is located within your infrastructure. We secure the endpoint with a self-signed certificate for encrypting the connection traffic. For reference, we show how that’s done using an Azure VM from the UI Portal.

Go to https://portal.azure.com/#home and select the Virtual Machines Icon from the list:
Fig 1. Creating a new Virtual Machine

Click on the +Add button in the next screen:

Fig 2. Add button

You will be presented with a new VM form. Make sure you fill in the necessary information, and open ports 80, 443 and 22 on the machine:

Fig 3. New VM form

Continue through all the steps on the form and click on Review & Create. It will perform a validation step to make sure you did not miss anything. After that, click on Create and wait until the machine is provisioned:

Fig 4. New VM form review

Once it completes, the screen will give you a link to go to that resource. Click on the button and then find the DNS name configuration to give the server a good alias:

Fig 5. Deployment complete

Fig 6. DNS config

You need to give it a good name in order to specify it on the self-signed certificate that you create on the server. For the purpose of this demonstration, we use conjur.northeurope.cloudapp.azure.com. Click Save.

Fig 7. DNS name

Go to the Overview Screen and click on the Connect button to show the panel:

Fig 8. Connection panel

From the connection panel information, copy the SSH connection string and execute it in your console:

$ ssh [email protected]
The authenticity of host 'conjur.northeurope.cloudapp.azure.com (13.70.192.28)' can't be established.
ECDSA key fingerprint is SHA256:RlQwpLy5VPaCrG7LmkpYWi8p4gkcFzEpWAF7AnJgaT0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'conjur.northeurope.cloudapp.azure.com,13.70.192.28' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.18.0-1024-azure x86_64)

Now we install Docker and Docker-Compose so we can provision all the required software easily.

Install Docker

$ sudo apt update
$ sudo apt install apt-transport-https ca-certificates curl software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
$ sudo apt update
$ sudo apt install docker-ce
$ sudo systemctl status docker
$ sudo usermod -aG docker ${USER}

Logout and login again to the shell.

Install Docker-compose

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ docker-compose --version

Install Conjur

Create a folder for the deployment files:

$ mkdir conjur && cd conjur

Create a default.conf file for the nginx configuration:

cat << ENDL > default.conf
server {
  listen 80;
  return 301 https://conjur$request_uri;
}
server {
  listen 443;
  server_name localhost conjur.northeurope.cloudapp.azure.com;
  ssl_certificate /etc/nginx/nginx.crt;
  ssl_certificate_key /etc/nginx/nginx.key;

  ssl on;
  ssl_session_cache builtin:1000 shared:SSL:10m;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
  ssl_prefer_server_ciphers on;

  access_log /var/log/nginx/access.log;

  location / {
    proxy_pass http://conjur:3000;
  }

  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
    root /usr/share/nginx/html;
  }
}

ENDL

Here we made sure to specify the FQDN for our server, as emphasized in bold.
Next, we create the self-signed certificates for the Nginx Proxy. Make sure you specify the full FQDN of your server on the Common Name entry:

$ openssl req -x509 -sha256 -newkey rsa:2048 -keyout cert.key -out cert.crt -days 1024 -nodes
Generating a RSA private key
.....................................+++++
...............+++++
writing new private key to 'cert.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IR
State or Province Name (full name) [Some-State]:Limerick
Locality Name (eg, city) []:Limerick
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Teckro
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:conjur.northeurope.cloudapp.azure.com
Email Address []:

After that step, create the docker-compose.yml file that will describe the Conjur Server:

$ cat << ENDL > docker-compose.yml
version: '2'
services:
  postgres:
    image: postgres:9.3

  conjur:
    image: cyberark/conjur:latest
    command: server -a cucumber
    environment:
      PORT: 3000
      DATABASE_URL: postgres://postgres@postgres/postgres
      CONJUR_DATA_KEY: "W0BuL8iTr/7QvtjIluJbrb5LDAnmXzmcpxkqihO3dXA="
    ports:
      - "3000:3000"
    depends_on: [ postgres ]

  proxy:
    image: nginx:1.13.6-alpine
    container_name: nginx_proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./default.conf:/etc/nginx/conf.d/default.conf:ro
      - ./cert.key:/etc/nginx/nginx.key:ro
      - ./cert.crt:/etc/nginx/nginx.crt:ro
    depends_on: [ conjur ]

  client:
    image: cyberark/conjur-cli:5
    entrypoint: sleep
    command: infinity
    environment:
      CONJUR_APPLIANCE_URL: https://proxy
      CONJUR_ACCOUNT: cucumber
      CONJUR_AUTHN_LOGIN: admin

ENDL

The Nginx Proxy listens to both ports 80 and 443, and it will proxy all the requests to https://conjur:3000

Next, run docker-compose up and observe the logs in the console. Look for a string named:

API key for admin:

and save it somewhere safe. This is the admin account key to interact with the REST API:

$ docker-compose up

It may take a while once the ports are propagated, but after a few minutes you should be able to visit the hostname and see the Conjur Welcome Page.

Now that we have the Conjur Server started, let’s try to install the Conjur-CLI tool to interact with the server from within the local machine.

First, on your local machine, install Conjur-CLI:

$ gem install conjur-cli

Run the init option to configure some defaults:

$ conjur init
Enter the URL of your Conjur service: https://conjur.northeurope.cloudapp.azure.com

SHA1 Fingerprint=40:41:5C:45:43:5B:E1:CF:9B:57:10:89:F4:CE:60:A1:2F:BE:FA:7E

Please verify this certificate on the appliance using command:
openssl x509 -fingerprint -noout -in ~conjur/etc/ssl/conjur.pem

Trust this certificate (yes/no): yes
Enter your organization account name: cucumber
Wrote certificate to /Users/itspare/conjur-cucumber.pem
Wrote configuration to /Users/itspare/.conjurr

The server is ready. With the last successful command, we can now focus on writing our infrastructure policies for multi-cloud deployments.

Step 2 – Design and Load Policies

Now that we have set up the server, let’s see what strategies we can use to handle secrets in a multi-cloud environment. Generally, when we work with multiple-cloud or hybrid-cloud providers, we need a plan for our configuration and for our credentials, in a way that it can be interchanged and not tied to a particular vendor.

To effectively do that, we need to understand how Policies work in Conjur. First, we have the root policy, which acts as the top policy for that account. It can contain all the users, groups, hosts and variables required to describe our organization. Below that root policy we assign branch policies, which are more specific for our deployments and types of secrets we manage.

Within the root policy we need to define an initial set of admin users and groups for the whole organization, only. We then use branches to refine that access and permissions for each defined variable.

The key thing to understand here is the layering of the branches and how to use different policies for different situations. We can categorize our first set of policies based on environments; for example, Development or Production environment. Then, each application has its own policies that control the access of secrets and permissions.

For our case, it makes sense to have a branch that can be used for each infrastructure provider; for example, an AWS policy and an Azure policy that sits on top of some other branches. We use those policies as sort of a namespace parameter for assigning branches into them. That way we can refer to them as:

cucumber:policy:aws/develop
cucumber:policy:azure/production
cucumber:policy:ds1/qa
cucumber:group:ds1/qa/testers
cucumber:group:azure/develop/developers
cucumber:group:aws/production/admins

You can use any Infrastructure naming conventions, according to your specific needs.

Let’s create our root policy first. We use it to define the admin group, which is the owner of the system policies, environments and infrastructures.

$ cat << ENDL > root.yml
# root.yml
- !group admin
- !permit
  role: !group admin
  privileges: [ read, create, update ]
  resource: !policy root
ENDL

Log in as admin and load the policy as root:

$ conjur authn login -u admin
Please enter admin's password (it will not be echoed):
Logged in

$ conjur policy load root root.yml
Loaded policy 'root'
  {
    "created_roles": {
  },
    "version": 1
  }

After each step it’s good practice to view the list of loaded objects:

$ conjur list
[
"cucumber:policy:root",
"cucumber:group:admin"
]

Next, look at the system policies. If using a custom authenticator such as AWS IAM or Kubernetes, you can define a policy. For example:

$ cat << ENDL > auth-ldap.yml
- !policy
  id: conjur
  owner: !group admin
  body:
    - !policy authn-ldap
    - !policy authn-iam/2884274272
ENDL

Next, define the users and groups that will be used to assign permissions and roles for each resource. Let’s create 4 groups and 4 users:

$ cat << ENDL > users.yml
- !group
  id: security_admin
  owner: !user admin

- !group
  id: developers
  owner: !group security_admin

- !group
  id: developers-admin
  owner: !group security_admin

- !group
  id: operations-admin
  owner: !group security_admin

# list the users
- !user
  id: alex.gavin
  owner: !group security_admin

# list the users
- !user
  id: mike.utah
  owner: !group security_admin

# list the users
- !user
  id: lisa.moran
  owner: !group security_admin

# list the users
- !user
  id: rory.hunter
  owner: !group security_admin

# add users to groups
- !grant
  role: !group developers-admin
  members:
    - !user mike.utah

- !grant
  role: !group operations-admin
  members:
    - !user lisa.moran

- !grant
  role: !group developers
  members:
    - !member
      role: !group developers-admin
    - !user rory.hunter
ENDL

Specific entitlements will be assigned for each user in another policy. When you load the policy under root, a list of keys for each user created is returned:

$ conjur policy load root users.yml
Loaded policy 'root'
{
"created_roles": {
"cucumber:user:alex.gavin": {
"id": "cucumber:user:alex.gavin",
"api_key": "jc3cna111zbfwz7rr962v8fn7x1g9tcz050rwkkypdgvq32maxar"
},
"cucumber:user:mike.utah": {
"id": "cucumber:user:mike.utah",
"api_key": "1mmrm3534vnddz22c4dys32bsaf83m0hgdj2x2padp15cp77w1xzyz1p"
},
"cucumber:user:lisa.moran": {
"id": "cucumber:user:lisa.moran",
"api_key": "2r48vf11c2sd78h8554v2zcc0c221e04jp5g40chrrycvc1kc4d6b"
},
"cucumber:user:rory.hunter": {
"id": "cucumber:user:rory.hunter",
"api_key": "bxarfc36hh36vjvk2c761zkvd1s2qpb73kea1be9j5xc41jzxm8h"
}
},
"version": 2
}

Another categorization is needed for the different cloud or on-prem infrastructure. Let’s create that now:

$ cat << ENDL > providers.yml
- !policy
  id: aws
  owner: !group security_admin

- !policy
  id: azure
  owner: !group security_admin
ENDL

$ conjur policy load root providers.yml

Create policies for two environments – develop and production. They can either be loaded using the same file, or a separate one for each environment under each provider:

$ cat << ENDL > environments.yml
- !policy
  id: develop
  owner: !group /developers-admin

- !policy
  id: production
  owner: !group /operations-admin
ENDL

$ ~/.gems/bin/conjur policy load azure environments.yml
$ ~/.gems/bin/conjur policy load aws environments.yml

Let’s review what we have now:

$ ~/.gems/bin/conjur list
[
...
"cucumber:policy:aws",
"cucumber:policy:azure",
"cucumber:policy:aws/develop",
"cucumber:policy:aws/production",
"cucumber:policy:azure/develop",
"cucumber:policy:azure/production"
]

Note
Multiple branches can be added under the same Layer. For example, if we were to include Amazon STS credentials, then we could use the following policy:

$ cat < sts.yml
---
- !policy
  id: sts
  body:
    - !policy
      id: blog-app
      body:
    # Secret Declarations
        - &secrets
          - !variable accessKey
          - !variable secretAccessKey
 
# === Layer for Automated Secret Access ===
    - !policy
      id: hosts
      annotations:
        description: Layer & Host Factory for machines that can read secrets
      body:
        - !layer
        - !host-factory
          layer: !layer

ENDL

We load them under AWS:

$ conjur policy load aws sts.yml

After that, we need to include the necessary permissions in the entitlements.yml file.

- !grant
  role: !group developers
  member: !layer aws/sts/hosts
- !permit
  role: !group developers
  privileges: [ read, execute ]
  resource: !variable aws/sts/blog-app/accessKey
- !permit
  role: !group developers-admin
  privileges: [ read, execute, update ]
  resources: !variable aws/sts/blog-app/accessKey
- !permit
  role: !group developers
  privileges: [ read, execute ]
  resources: !variable aws/sts/blog-app/secretAccessKey
- !permit
  role: !group developers-admin
  privileges: [ read, execute, update ]
  resources: !variable aws/sts/blog-app/secretAccessKey

And we load them under root:

$ conjur policy load entitlements.yml
Loaded policy 'root'
{
  "created_roles": {
  },
  "version": 4
}

The above permissions will allow the developers group to read the STS credentials, but the developers-admin to update them.
If you have policies per regions – for example for EU, US, ASIA – you could load them for each provider, like this:

“cucumber:policy:aws/eu/develop”,
“cucumber:policy:aws/us/production”

Writing policies should ideally match the hierarchy profiles of your organization, for consistency and convention.
Here is how to add another categorization for frontend and backend roles:

$cat << ENDL > sides.yml
- !policy
  id: backend

- !policy
  id: frontend
ENDL

$ conjur policy load azure/develop sides.yml
$ conjur policy load aws/develop sides.yml

$ conjur list
[
...
  "cucumber:policy:aws/develop/backend",
  "cucumber:policy:aws/develop/frontend"
  "cucumber:policy:azure/develop/backend",
  "cucumber:policy:azure/develop/frontend"
]

At the end, you need policies for your applications. Create a blog-app.yaml file that represents a policy for a blog application that connects to a database using a connection URL:

$cat << ENDL > blog-app.yml
- !policy
  id: blog-app
  body:
    - &variables
      - !variable db-connection-url

    - !group secrets-blog-app

# secrets-blog-app can read and execute
    - !permit
      resource: *variables
      privileges: [ read, execute ]
      role: !group secrets-blog-app
ENDL

$ ~/.gems/bin/conjur policy load aws/develop/backend blog-app.yml
$ ~/.gems/bin/conjur policy load azure/develop/backend blog-app.yml

$ conjur list
[
...
"cucumber:variable:azure/develop/backend/blog-app/db-connection-url",
"cucumber:group:azure/develop/backend/blog-app/secrets-blog-app",
"cucumber:policy:aws/develop/backend/blog-app",
"cucumber:variable:aws/develop/backend/blog-app/db-connection-url",
"cucumber:group:aws/develop/backend/blog-app/secrets-blog-app"
]

As you can see, we can verify the list of objects that are within the proper namespace.

To apply permissions to each group and user, a separate file is needed for user entitlements.yml that are loaded from the root account. This requires a good amount of planning and checking in order to apply the right permissions for each group, user and layer.

A typical role assignment:

$cat << ENDL > entitlements.yml
- !grant
  role: !group azure/develop/backend/blog-app/secrets-blog-app
  member: !group developers

- !grant
  role: !group aws/develop/backend/blog-app/secrets-blog-app
  member: !group developers
ENDL

$ conjur policy load root entitlements.yml

The above policy allows the developers group to be a member of the azure/develop/backend/blog-app/secrets-blog-app and aws/develop/backend/blog-app/secrets-blog-app groups, and grants them access to the secrets-blog variables to connect to the database. This granular level of permissions is leveraging the true power of RBAC in practice.

Verify that this holds by issuing the following command; Yes means that the particular resource has permissions to access the variable:

$ conjur check group:developers variable:azure/develop/backend/blog-app/db-connection-url
Yes

All the permissions of a resource can be listed by issuing:

$ conjur show variable:azure/develop/backend/blog-app/db-connection-url
{
"created_at": "2019-07-31T15:36:09.262+00:00",
"id": "cucumber:variable:azure/develop/backend/blog-app/db-connection-url",
"owner": "cucumber:policy:azure/develop/backend/blog-app",
"policy": "cucumber:policy:azure/develop/backend",
"permissions": [
{
"privilege": "read",
"role": "cucumber:group:azure/develop/backend/blog-app/secrets-blog-app",
"policy": "cucumber:policy:azure/develop/backend"
},
{
"privilege": "execute",
"role": "cucumber:group:azure/develop/backend/blog-app/secrets-blog-app",
"policy": "cucumber:policy:azure/develop/backend"
}
],
"annotations": [

],
"secrets": [

]
}

Now that policies are defined, the last step is to populate the variables with secrets and try to access them from the clients.

Step 3 – Defining and Consuming Secrets

With policies defined describing access controls, let’s see how to store and retrieve sensitive variables across the multi-cloud infrastructure.
Nearly everything can be stored as a string or binary in Conjur. As a representative example, we will store the db-connection-url for our Blog application under Azure. This example could be extended to cover all sorts of secrets and sensitive values, from SSL certificates, CI/CD credentials, account credentials (such as AWS, GCE or Azure), and more.
To store the db-connection-url:
Logout from admin and login as mike.utah – a member of developers-admin.

$ conjur authn logout
  Logged out
$ conjur authn login -u mike.utah
  Please enter mike.utah's password (it will not be echoed):
  Logged in

When we login with that user and issue a list command, all the variables that user can read or update are shown:

$ conjur list
"cucumber:variable:aws/sts/blog-app/accessKey",
"cucumber:variable:aws/sts/blog-app/secretAccessKey",
"cucumber:policy:aws/develop",
"cucumber:policy:azure/develop",
"cucumber:policy:azure/develop/backend",
"cucumber:policy:azure/develop/frontend",
"cucumber:policy:azure/develop/backend/blog-app",
"cucumber:variable:azure/develop/backend/blog-app/db-connection-url",
"cucumber:group:azure/develop/backend/blog-app/secrets-blog-app",
"cucumber:policy:aws/develop/backend",
"cucumber:policy:aws/develop/frontend",
"cucumber:policy:aws/develop/backend/blog-app",
"cucumber:variable:aws/develop/backend/blog-app/db-connection-url",
"cucumber:group:aws/develop/backend/blog-app/secrets-blog-app"
]

We are only interested in the cucumber:variable:azure/develop/backend/blog-app/db-connection-url variable. Let’s store a secret there:

$ conjur variable values add azure/develop/backend/blog-app/db-connection-url "mysql://blog_root:[email protected]/blog_db"
Next, check it’s value, making sure to url escape the variable value first:
$ conjur variable value azure%2Fdevelop%2Fbackend%2Fblog-app%2Fdb-connection-url
mysql://blog_root:[email protected]/blog_db

Accessing secrets from our application should be a piece of cake now. We might need to add additional machine identity users with a read-only access for those secrets. I leave it for the reader to do that.
For reference, Conjur provides APIs for the following languages:
● .NET
● Go
● Java
● Ruby

 

A REST API is also available. The detailed docs are located on this page, and we strongly recommend you explore all the options there.
Overall, we can say that the most critical part of setting up a complete secrets solution with Conjur lies in the effective application of policies and permissions. After that step is finished, the process becomes streamlined and secure by default. To learn more about Conjur, visit the official Website.

 

Continue The Conversation!

If you enjoyed this project or have questions join me on the CyberArk Commons. You can network with other engineers, ask for help, and contribute to the future of Conjur OSS.