Since our first release of the Conjur Open Source Suite (OSS) aimed at helping development teams make the most of our secrets management service, we have been eager to share some of our favorite techniques and technologies used along the way. And while we’re realizing our goal of reducing the developer burden by automating everything that we can – we’re continuing to find new ways to simplify and strengthen DevOps security. Through this post, we hope that you can use some of our learning in your own work to save both time and headaches.
So without further ado, let’s take a look at some of the key components that create our suite releases:
Changelog Standards
Consistency and simplicity go hand-in-hand. So, the first thing we focused on was standardizing a format for all of the suite’s dependency changelogs. This would allow portably parsed structured data across repos and enforce consistency in presentation. For this, we used the widely known keep-a-changelog format. This format provides us with all the necessary release documentation that we could then extract and use in our suite artifacts.
By adding a small script that uses our parse-a-changelog gem to all suite repo CI/CD builds, we enforce the structure of those files to retain conformance to the format over time. The increased rigidity in the format has also made our changelogs easier to work with as it removed a lot of ambiguity related to its structure.
Changelog Parsing
With the changelogs in a standardized format, we could aggregate them easily by applying a bit of abstract syntax tree (AST) parsing to extract objects of interest. We use the standard ast Golang library along with some code to extract the relevant sections of the changes from each repository’s CHANGELOG.md
that we then splice together for the final suite documentation. This allows us to compose any structure in the suite documentation that we need and then stitch it via templates.
Golang
If you need code that is fast, portable, ubiquitous, and extensible, Golang is an implementation language no-brainer. For us, it was an easy choice, particularly considering our team’s solid language competency, great standard library, and clearly defined objects. It’s worth noting that testing Golang code can be somewhat painful versus competitors, but that is a small price to pay for the benefits it provides.
Suite Configuration
For configuration to represent which repos are pulled together in the suite, we went with the most obvious standard: YAML. While not perfect, it is the lingua franca of the DevOps world. Its use is so prevalent that any other choice would have added maintenance burdens, which we wanted to avoid. We composed the YAML structure to match our parsing objects 1:1 so that we could easily ingest it in Golang via something as simple as yaml.Unmarshal(yamlFile, &repoConfig). This allows us flexibility to expand the structure as needed and also provides a direct object-relational mapping.
GitHub APIs
As with most current open source projects, GitHub was the natural place to host our suite components, encourage collaboration and minimize version control issues. By leveraging Github’s APIs, we are able to pull all the relevant data (e.g. changelogs, branches, releases, etc.) for the suite documentation easily. The API common usage is obvious, well documented, and easily testable – making for a very pleasant and productive development experience. We do hit their API limits off-and-on, but generally, those limits are quite reasonable even for our rather heavy use case.
GitHub Actions
Tying all of these technologies together requires some orchestration. We do have an internal Jenkins-based CI/CD pipeline, but we also needed an open, transparent way for contributors to examine and improve upon it.
Based on what we knew about GitHub actions, we decided to give it a spin, using those pipelines for practically all of the automation. This proved to be a great decision for us. With GitHub actions, we’re automating workflows such as:
– Running unit tests on the master branch
– Running the end-to-end tests on all pull requests and master branch (including running Helm charts in KIND!)
– Nightly updates of unreleased suite changes to the wiki page
– Creating draft releases when a new tag is pushed
– Opening a pull request automatically when a new release is made to bake in the suite versions into the repo
Automating these tasks has helped us save a ton of time. Today, our manual work is typically limited to updating the configuration file and reviewing generated files – our GitHub actions take care of the rest, allowing us to focus on new ways to help the community create and secure great software.
Why work harder when you can work smarter? 😀
Epilogue
While we don’t expect our readers to use of all of the same tooling, we hope it serves as inspiration for your own projects. And though this isn’t an exhaustive list, we consider these pieces to be critical to our project’s success so far.
If you feel adventurous, visit our project’s code on GitHub to see what other tools and methods our team is using.
PS: We ❤️ contributions so feel free to open an issue or open pull requests if you find something that can be improved!
Until Next Time!
CyberArk Community and Integrations Team
Srdjan Grubor is a Software Engineer at CyberArk where he is building the next-generation digital security products. Srdjan is the author of “Deployment with Docker” book, was one of the first people to receive a Docker Certified Associate certification, and has worked on Linux systems at scale for over a decade. He enjoys breaking things just to see how they work, tinkering, and solving challenging problems.