Stop Bashing Bash

A New Open Source Library & Pattern for Sharing Bash

Published October 31st, 2019 by Hugh Saunders

Admit it, you still write bash. Maybe a lot of bash. We all think we can avoid it, but we still end up using it to tie our various tools together. For most of our tools, we’re used to searching code libraries like RubyGems or PyPi for shared packages we can use. But, writing bash often lands us on Google or StackOverflow instead. What if the common things we all do in bash were available in a public, shareable, well-tested library?

Bash-lib is an open source library of reusable functions for the bash scripting language. We created it to reduce code duplication and improve the reliability of the scripts we use in our continuous integration (CI) pipelines. These scripts are critical to the software development process, but are often untested. With bash-lib we are able to write shareable well-tested code that can benefit many projects. We have encountered situations in the past where bash errors lead to incorrect CI results – but this could have been avoided if we had appropriate unit testing in place.

Bash-lib is not a complete library of bash functions; in fact, it’s almost empty. Our aim is to create a maintainable space that will remain easy to use as developers contribute their own bash functions. To support the maintainability goal, early work has addressed testing and documentation.

Testing

All functions in bash-lib are tested with BATS (Bash Automated Testing System). BATS is a unit test framework and each test aims to cover as small a unit of functionality as possible.  There is also a lint test to ensure that all top level library functions have a BATS test.  Scripts are all also statically analyzed using shellcheck and gitleaks.

Documentation

Libraries based on a language rather than a specific task can become disorganized if multiple similar functions are added. To maintain order and reduce duplication, all public (top level) library functions are listed and documented in the project README. This flat list helps users to find available functions. The documentation also help contributors to detect duplicates. A lint test ensures that new functions are added to the README.

How to Determine if a Function is Suitable for bash-lib

Features of a good library function include:

  1. Focused: A deploy-all-the-things function is unlikely to be useful in a library as its unlikely that other users will want exactly the same deployment. A small function that achieves a specific task would be more useful, e.g. a function that creates ansible venv.
  2. All inputs as arguments: Functions that read from the filesystem or environment are likely to cause problems when re-used unless the prerequisites are documented and agreed. Functions that do require other inputs should fail with descriptive error messages when they are missing.
  3. Duplicate functionality: Functions that appear copy/pasted in multiple repos or continue to reappear as different functions with the same purpose would be a great addition to bash-lib.

How Can I Use Bash-Lib in My Project?

  1. Add the submodule: 
    $ git submodule add [email protected]:cyberark/bash-lib bash-lib
  2. Write a script that sources the relevant libraries and uses a function. For example, the following snippet uses the repo_root function to determine the git repo root above the current directory: 
    #!/bin/bash
# Initialize bash-lib
. ./bash-lib/init
# Source the git library
. ${BASH_LIB_DIR}/git/lib

# Go down a few dirs
mkdir -p a/b/c
cd a/b/c
# Determine the root of the current repo
repo_root

Why Open Source bash-lib at This Stage?

Often projects are kept closed until they reach an MVP stage. However we decided to publish this repo early so that any interested parties can influence its structure early on. While we’re not expecting a Kubernetes-sized community to build up around bash-lib, shell was the 6th most popular language in Stack Overflow’s 2019 Developer Survey, so there are a large number of bash hackers out there! If a few of us share some great functions, we’ll have saved each other some time 🙂

Open Source at CyberArk

At CyberArk, we benefit from many great open source projects from the Linux kernel right up through the stack to Kubernetes and Golang. We have initiated open source projects such as Conjur and Secretless Broker and actively contribute to projects we use. We are always building more products in the open within CyberArk Commons, so look out for future announcements like this!

Bash-Lib Future Work

To get a sense of the project direction, take a look at our backlog of planned updates:

  • Introduce a doc-comment system so that the function index can be automatically generated. One possibility is tomdoc.sh. (Issue 11)
  • Fill out the stub libraries by adding functions.
  • BATS is currently integrated into bash-lib as a subtree. We may switch to submodules, please let me know if you have a strong opinion either way. (Part of Issue 7)

Want to Contribute?

Sign the contributing agreement and create a PR. For details see the contributing guide.

Bash Links

Join the Conversation on CyberArk Commons

This project and other open source projects are part of the CyberArk Commons community.  If you like what you see here, have your own project to share, or would like to discuss this project with our community, join us on Discuss.CyberArkCommons.org.

Comments are closed.