Litmus test for CI/CD pipelines

In chemistry, litmus paper is a thin strip of paper treated with a special red or blue dye that is used to determine the acidity of a substance. An acid turns blue litmus paper red, a base turns red litmus paper blue, and neutral substances don't affect the paper's color at all.

The way litmus paper determines acidity is by measuring a substance's pH level. If a pH level is higher than 8, it's acidic; if it's below 5, it's basic; and if it's between 5 and 8, it's neutral. Similarly, the CI/CD litmus test helps you measure the CI/CD level of your pipeline.

To test whether your pipeline is fully CI/CD

1. Start with a score of 0.

2. Answer each of the following questions, and add 1 to your score for each time you answer yes:

   • Do our repositories each have exactly one main branch that is used to deploy to environments?

   • Do we commit code to the main branch frequently and avoid having long-running feature branches?

   • Does our pipeline have a single point of entry? In other words, does our pipeline pull the code from each repository exactly one time?

   • Do we have more than one deployment environment?

   • When the pipeline is not running, are our upper and lower environments generally in sync?

   • Do we run tests on code before deploying?

   • Do we run tests on an environment before promoting to the next environment?

   • Does our pipeline do a full rollback and exit after a failure?

   • Does our pipeline restart from the first step when recovering from a failure?

   • Do we follow the same process to fix bugs in production that we do to release features to production?

   • Do we use some form of infrastructure as code (IaC) templates to deploy code?

3. Answer each of the following questions, and add 1 to your score for each time you answer no:

   • Do we ever deploy directly into a deployment environment from branches other than the main branch?

   • Do we ever deploy directly from any branch to an upper or production environment?

   • Do we often find bugs in upper environments that were not present in lower environments?

   • Do we ever bypass lower environments during a deployment?

   • Do we wait until a scheduled release time to deploy to production?

   • Do we regularly make updates in the production environment's console?

   • Are there any manual deployment steps that must be done in the production environment's console in order to complete deployment?

   • Does more than one person have write access to the production environment?

   • Do more than five people have write access to the production environment?

4. Divide your score by 2. This is your pipeline's CI/CD score.

5. Compare your pipeline's CI/CD score to the following table to determine your pipeline's CI/CD level.

   CI/CD score	CI/CD level
   9.5 or above	Fully CI/CD
   8–9	Mostly CI/CD
   5–7	Neutral
   Below 5	Not CI/CD

If you scored below 8, we recommend that you set a goal to move gradually toward the next level. When that goal is achieved, then product stakeholders should assess if and when a new goal should be set. The intention of this exercise is not necessarily to advocate for a change to your pipeline, but rather to bring awareness of what a fully CI/CD deployment process looks like and where your pipelines currently sit on that spectrum.