Decentralized Governance

As your organization grows and establishes more code-driven business processes, one challenge it could face is the necessity to scale the engineering team and enable it to work efficiently in parallel on a large and diverse codebase. Additionally, your engineering organization will want to solve problems using the best available tools.

Decentralized governance is an approach that works well alongside microservices to enable engineering organizations to tackle this challenge. Traffic lights are a great example of decentralized governance. City traffic lights may be timed individually or in small groups, or they may react to sensors in the pavement. However, for the city as a whole, there is no need for a primary traffic control center in order to keep cars moving. Separately implemented local optimizations work together to provide a city-wide solution. Decentralized governance helps remove potential bottlenecks that would prevent engineers from being able to develop the best code to solve business problems.

When a team kicks off its first greenfield project it is generally just a small team of a few people working together on a common codebase. After the greenfield project has been completed, the business will quickly discover opportunities to expand on their first version. Customer feedback generates ideas for new features to add and ways to expand the functionality of existing features. During this phase, engineers will start growing the codebase and your organization will start dividing the engineering organization into service-focused teams.

Decentralized governance means that each team can use its expertise to choose the best tools to solve their specific problem. Forcing all teams to use the same database, or the same runtime language, isn’t reasonable because the problems they’re solving aren’t uniform. However, decentralized governance is not without boundaries. It is helpful to use standards throughout the organization, such as a standard build and code review process because this helps each team continue to function together.

Source control plays an important role in the decentralized governance. Git can be used as a source of truth to operate the deployment and governance strategies. For example, version control, history, peer review and rollback can happen through Git without needing to use additional tools. With GitOps, automated delivery pipelines roll out changes to your infrastructure when changes are made by pull request to Git. GitOps also makes use of tools that compares the production state of your application with what’s under source control and alerts you if your running cluster doesn’t match your desired state.

The following are the principles for GitOps to work in practice:

1. Your entire system described declaratively

2. A desired system state version controlled in Git

3. The ability for changes to be automatically applied

4. Software agents that verify correct system state and alert on divergence

Most CI/CD tools available today use a push-based model. A push-based pipeline means that code starts with the CI system and then continues its path through a series of encoded scripts in your CD system to push changes to the destination. The reason you don’t want to use your CI/CD system as the basis for your deployments is because of the potential to expose credentials outside of your cluster. While it is possible to secure your CI/CD scripts, you are still working outside the trust domain of your cluster which is not recommended. With a pipeline that pulls an image from the repository, your cluster credentials are not exposed outside of your production environment.

The following are the key factors from the twelve-factor app pattern methodology that play a role in enabling decentralized governance:

• Dependencies – Decentralized governance allows teams to choose their own dependencies, so dependency isolation is critical to make this work properly.

• Build, release, run – Decentralized governance should allow teams with different build processes to use their own toolchains, yet should allow releasing and running the code to be seamless, even with differing underlying build tools.

• Backing services – If each consumed resource is treated as if it was a third-party service, then decentralized governance allows the microservice resources to be refactored or developed in different ways, as long as they obey an external contract for communication with other services.

Centralized governance was favored in the past because it was hard to efficiently deploy a polyglot application. Polyglot applications need different build mechanisms for each language and an underlying infrastructure that can run multiple languages and frameworks. Polyglot architectures had varying dependencies, which could sometimes have conflicts.

Containers solve these problems by allowing the deliverable for each individual team to be a common format: a Docker image that contains their component. The contents of the container can be any type of runtime written in any language. However, the build process will be uniform because all containers are built using the common Dockerfile format. In addition, all containers can be deployed the same way and launched on any instance since they carry their own runtime and dependencies with them.

An engineering organization that chooses to employ decentralized governance and to use containers to ship and deploy this polyglot architecture will see that their engineering team is able to build performant code and iterate more quickly.