What is Secretless?
Connecting to APIs has become a standard part of application development. As developers, we regularly use APIs to enhance our applications. Unfortunately, this benefit doesn’t come without risks. The major risk being: how can I be sure my application won’t leak the secrets it uses to authenticate with APIs? Normally, if you want to connect to an API, you will make a request that includes your credentials:
curl --request GET \ --header ‘Authorization: Bearer ’ --url 'https://api.twitter.com/1.1/search/tweets.json?q=nasa&result_type=popular'
But, leaving your access credentials or secrets in your terminal history – and worst-case scenario, the code you just pushed to your public repo – is just asking for trouble. A threat actor can easily get their hands on them, putting your environment at risk.
You want to move fast so you can deliver value to your end users, and it’s hard to manage extra secrets management responsibilities that could slow things down. But you also recognize that it’s a problem if your application inadvertently logs or exposes your credentials.
Secretless Broker solves this problem.
Secretless Broker (or Secretless for short) is an open source tool that abstracts away the notion that developers must be the ones managing their API keys, tokens, and other credentials either by connecting to a vault or setting up their own env
file. Instead of the request above, we can now have a request that looks like this:
http_proxy=localhost:8061 \ curl --request GET \ --url 'api.twitter.com/1.1/search/tweets.json?q=nasa&result_type=popular'
This request still works the exact same way, but it includes no credentials. Secretless will add the authentication headers for you or replace these headers in the original request like magic (but, we promise, it really isn’t!).
Most target services such as APIs or databases require their own unique Secretless connector to connect to Secretless. The Secretless team has tried to make it as easy as possible to extend the project and write new connectors – check out the SDK docs to learn more.
For API target services, Secretless comes with support for:
- AWS
- Conjur
- Any API that uses Basic Authentication
Though there are a lot of APIs that just use basic auth, these three connectors alone don’t cover every API you’d possibly want to connect with. This means with these three connectors alone, it’s nonzero effort to use Secretless with your own custom API. And though the SDK makes it possible to write a custom connector to match any authentication flow, we’ve not yet taken advantage of a really important point: most API authentication methods are pretty similar.
Enter the Generic HTTP Connector
The Generic HTTP connector enables you to use Secretless with a wide array of HTTP-based services without having to write new Secretless connectors. Instead, you can modify your Secretless configuration YAML to specify the header structure the HTTP service requires to authenticate. This makes authenticating most HTTP requests with Secretless easy and flexible. To demonstrate, we developed example configurations that work for over 20 common target APIs. Configurations include:
- Twitter API
- Tableau API
- GitHub API
- Google Maps APIs
- …and more!
What we learned from writing 20+ Generic HTTP configurations for Secretless
Before writing a Secretless configuration for the API you need, make sure to review the examples to see which API configurations have already been built. The documentation includes the API versions, instructions for how to authenticate to that API, and examples of running the configuration locally for over 20 configurations.
When writing a connector for a specific API, the most important thing to understand before starting is how the API authorizes requests. Currently, Secretless’ HTTP Generic Connector supports the following authorization methods:
- Authorization Headers
- Basic Auth
- OAuth 1.0
- OAuth 2.0
- Service Specific Headers (X-Service-Token, X-API-Key, token)
- Query Parameters
To understand how to get started writing a configuration to get the generic HTTP connector working with a new API, let’s walk through a specific example.
Writing a generic HTTP connector config for GitHub’s V3 API
To start writing a generic HTTP connector configuration that works with GitHub’s V3 API, we first needed to look at how GitHub authenticates requests sent to their API. GitHub supports both basic authentication and OAuth 2.0, but we’ll use the OAuth 2.0 authentication method in this example because it’s what GitHub recommends.
Since the request expects a static OAuth Token in the header, this is a perfect scenario for the Generic HTTP Connector. Notice the header format; we’ll need to remember that when we fill out our configuration later.
After establishing how the API authorizes requests, we move on to the next step of storing the needed credentials in one of the supported secret providers. In our example, we’ll be storing the credentials in the OSX keychain. If you’d like to try this too, you can store the GitHub token using conceal by running:
conceal set github/api_token
Now that we understand the API’s authentication scheme and we’ve stored our credentials, we can build the Secretless configuration file, or secretless.yml
. We’ll do this following the generic HTTP connector standard, which requires two important steps in this case:
- Under the
credentials
key in the Secretless configuration file, include your credentials and where to fetch them. - Under the
config
key in the Secretless configuration file, add aheaders
key for the OAuth 2.0 header we are mimicking. This is where we create the header the GitHub API is expecting with a variable for the token we’re retrieving:Authorization: token {{ .token }}
The resulting secretless.yml
file looks like this:
version: 2 services: github: connector: generic_http listenOn: tcp://0.0.0.0:8081 credentials: token: from: keychain get: summon#github/api_token config: headers: Authorization: token {{ .token }} forceSSL: true authenticateURLsMatching: - ^http[s]*\:\/\/api\.github.*
It’s worth noting that if your API supports multiple authentication modes, you can create a Secretless service configuration for each authentication method that you would like to incorporate. We could have added a second service that uses GitHub’s basic authentication scheme if we needed to use both modes. To see an example, visit our example generic HTTP connector configuration for the Twitter API.
Note also that the variables in the config
section match the keys of the retrieved secrets in the credentials
section. When Secretless is running, it will inject the Authorization
header, replacing the token
variable with the fetched value from the keychain
service.
Once the configuration is set, you can run Secretless, make your request, and watch the magic happen!
Sometimes, the generic HTTP connector isn’t a good fit…
The Generic HTTP Connector is great for interacting with APIs and applications that authenticate in a static way by passing API tokens in every request. The use case starts to get a little more complicated when the methods of authentication begin to deviate from this. To help you recognize when your API isn’t a good fit for the generic HTTP connector, we’ve written about a few more complicated use cases that we discovered.
Query Parameters
Originally, the Secretless Broker HTTP Generic Connector only supported injecting headers into a request.
After identifying that this would cause an issue with the Google Maps Web Services API, which used query parameters to send API keys with HTTP requests, support for injecting query parameters was added to the HTTP Generic Connector. You can now write your generic HTTP connector configuration so that Secretless will inject specific query parameters into your requests.
Once we added this feature, we were able to create a Google Maps Web Services connector configuration that supports injecting an API key securely via Secretless.
OAuth 1.0
Like query parameters, OAuth 1.0 was not supported by Secretless when we started this project. This made it impossible to authenticate with services such as Twitter’s OAuth 1.0a API.
The main reason OAuth 1.0 was not initially supported was the way the Authorization
header is created. Since OAuth 1.0 uses the secrets that Secretless protects to create a signature parameter for the header, we needed a way to create this header on Secretless’ side.
To fill this gap, we decided to add this functionality to the Generic HTTP Connector as an oauth1
key inside of the config
. Now this method of authentication is supported and Secretless will create the entire OAuth 1.0 Authorization
header for you. See our OAuth 1.0 docs for more information.
APIs That Use Custom CLIs
While there is a Docker Registry API configuration made with the Generic HTTP Connector, the Docker CLI is a whole other story. Since the Docker CLI authenticates by using locally stored credentials to retrieve an API key before making the authenticated request, the Generic HTTP Connector interferes in this handshake between the Docker daemon and the Docker Registry. The best work around for scenarios like this is the create a separate, application specific connector.
Most of the time, the generic HTTP connector is all you need
The Generic HTTP Connector is a simple solution to seamlessly add the security of Secretless Broker to your projects, without the trouble of creating a brand-new Secretless HTTP connector for each API-based service you are authenticating to. The Generic HTTP Connector makes it easier than ever to quickly authenticate to many APIs – today, we have 25 examples and counting of how to do this!
We hope these ideas help inspire you, but if you try them out and the Generic HTTP Connector doesn’t fit your needs, we’re here to help. You can also contribute to this project by adding more authentication methods, secret providers, or service configurations – come join the conversation in CyberArk Commons.
Mitchell Dodell
Mitchell Dodell was a software engineer intern at CyberArk during the summer of 2020. A rising senior at Brandeis University studying Computer Science and Business, Mitchell is an active member on campus where he organizes hackathons, consults with Israeli start-ups, and works at the Brandeis Undergraduate Admissions office. Be sure to follow him on LinkedIn to see what he’ll do next.
Jake Quilty
Jake Quilty is a software engineer intern at CyberArk. He is a Computer Science student at Wentworth Institute of Technology, with an interest in cyber security. Outside of work, Jake likes to participate in CTF competitions, hackathons, and is a big Boston sports fan. To see what project he’s working on now, check out his GitHub.