Use Serverspec for test-driven development of infrastructure code

Created by Sushant Jagdale (AWS)

Environment: PoC or pilot

Technologies: DevOps; Infrastructure; Hybrid cloud

AWS services: Amazon EC2; AWS CodeBuild; AWS CodeDeploy

Summary

This pattern shows you how to use Serverspec to use test-driven development (TDD) when writing infrastructure code on the Amazon Web Services (AWS) Cloud. The pattern also covers automation with AWS CodePipeline. TDD will focus attention on what the infrastructure code must do and sets a clear definition of done. You can use Serverspec to test infrastructure created by tools such as AWS CloudFormation, Terraform by HashiCorp, and Ansible.

Serverspec helps with refactoring infrastructure code. With Serverspec, you can write RSpec tests to check installation of various packages and software, run commands, check for running processes and ports, check file permission settings, and so forth. Serverspec checks whether your servers are configured correctly. You install only Ruby on your servers. You don't need to install any agent software.

Test-driven infrastructure provides the following benefits:

• Cross-platform testing

• Validation of expectations

• Confidence in your automation

• Infrastructure consistency and stability

• Fail early

You can use this pattern to run Serverspec unit tests for Apache software and check file permission settings during Amazon Machine Image (AMI) creation. An AMI will be created only if all the test cases pass. Serverspec will perform following tests:

• Apache process is running.

• Apache port is running.

• Apache configuration files and directories exist at certain locations, and so forth.

• File permissions are correctly configured.

Prerequisites and limitations

Prerequisites

• An active AWS account

• AWS CodeBuild

• AWS CodeCommit

• AWS CodePipeline

• A virtual private cloud (VPC) with a public subnet

• Installation of AWS Command Line Interface (AWS CLI) and Git

Product versions

• HashiCorp Packer version: 1.6.6

• Ruby version: 2.5.1 and later

• AWS CLI version: 1.18.185

Architecture

Target architecture

1. When you push the code to the CodeCommit repository, an Amazon CloudWatch Events event engages the CodePipeline. In the first stage of the pipeline, the code is fetched from CodeCommit.

2. The second pipeline stage runs CodeBuild, which validates and builds the Packer template.

3. As a part of the Packer build provisioner, Packer installs Apache and Ruby software. Then the provisioner calls a shell script that uses Serverspec to unit test the Apache process, port, files, and directories. The Packer post-processor writes a JavaScript Object Notation (JSON) file with a list of all the artifacts produced by Packer during a run

4. Finally, an Amazon Elastic Compute Cloud (Amazon EC2) instance is created using the AMI ID produced by Packer.

Tools

• AWS CLI – Amazon Command Line Interface (AWS CLI) is an open source tool for interacting with AWS services using commands in your command line shell.

• Amazon CloudWatch Events – Amazon CloudWatch Events delivers a near-real-time stream of system events that describe changes in Amazon Web Services (AWS) resources.

• AWS CodeBuild –  AWS CodeBuild is a fully managed build service in the cloud. CodeBuild compiles your source code, runs unit tests, and produces artifacts that are ready to deploy.

• AWS CodeCommit – AWS CodeCommit is a version control service hosted by Amazon Web Services. You can use CodeCommit to privately store and manage assets (such as documents, source code, and binary files) in the cloud.

• AWS CodePipeline – AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software. You can quickly model and configure the different stages of a software release process.

• HashiCorp Packer – HashiCorp Packer is a tool for automating the creation of identical machine images from a single source configuration.

• Serverspec – Serverspec runs RSpec tests to check server configuration. Serverspec uses Ruby, and you don't need to install agent software.

Code 

The code is attached. The code uses the following structure, with three directories and eight files.

├── amazon-linux_packer-template.json (Packer template)
├── buildspec.yaml (CodeBuild .yaml file)
├── pipeline.yaml (AWS CloudFormation template to automate CodePipeline)
├── rspec_tests (RSpec required files and spec)
│   ├── Gem-file
│   ├── Rakefile
│   └── spec
│       ├── apache_spec.rb
│       └── spec_helper.rb
└── scripts
    └── rspec.sh (Installation of Ruby and initiation of RSpec)

Epics

Configure AWS credentials

Task	Description	Skills required
Create an IAM user.	Create an AWS Identity and Access Management (IAM) user with programmatic and console access. For more information, see the AWS documentation.	Developer, Systems administrator, DevOps engineer
Configure AWS credentials.	On your local computer or in your environment, configure AWS credentials for the IAM user. For instructions, see the AWS documentation.	Developer, Systems administrator, DevOps engineer
Test your credentials.	To validate the configured credentials, run the following command. aws sts get-caller-identity --profile <profile>	Developer, Systems administrator, DevOps engineer

AWS CodePipeline

Task	Description	Skills required
Create a CodeCommit repository.	To create a CodeCommit repository, run the following command. aws codecommit create-repository --repository-name "<provide repository-name>" --repository-description "repository to unit test the infrastructure code"	Developer, Systems administrator, DevOps engineer
Write RSpec tests.	Create RSpec test cases for your infrastructure. For more information, see the Additional information section.	Developer, DevOps engineer
Push code to the CodeCommit repository.	To push the attached code to the CodeCommit repository, run the following commands. git clone <repository url> cp -R /tmp/<code folder>/ <repository_folder>/ git add . git commit -m"initial commit" git push	Developer, Systems administrator, DevOps engineer
Create the pipeline.	To create the pipeline, run the AWS CLI command that is in the Additional information section.	Developer, Systems administrator, DevOps engineer
Start the pipeline.	Commit code to the CodeCommit repository. Any commit to the repository will initiate the pipeline.	Developer, Systems administrator, DevOps engineer
Test the Apache URL.	To test the AMI installation, use the following URL. http://<your instance public ip>/hello.html The page will show a "Hello from Apache" message.	Developer, Systems administrator, DevOps engineer

Related resources

• HashiCorp

• HashiCorp Packer

• Serverspec

• Introduction to ServerSpec: What is Serverspec and how do we use it at Stelligent? (external blog post)

• Test-driven development of infrastructure code (external blog post)

• Image creation and testing with HashiCorp Packer and ServerSpec (external article)

Additional information

Write RSpec tests

The RSpec test for this pattern is located at <repository folder>/rspec_tests/spec/apache_spec.rb.

require 'spec_helper'


describe service('httpd') do
  it { should be_enabled }
  it { should be_running }
end


describe port(80) do
  it { should be_listening }
end


describe file('/etc/httpd/conf/httpd.conf') do
  it { should exist }
  it { should be_owned_by 'root' }
  it { should contain 'ServerName www.example.com' }
end


describe file('/etc/httpd/conf/httpd.conf') do
  its(:content) { should match /ServerName www.example.com/ }
end


describe file('/var/www/html/hello.html') do
  it { should exist }
  it { should be_owned_by 'ec2-user' }
end


describe file('/var/log/httpd') do
  it { should be_directory }
end


describe file('/etc/sudoers') do
  it { should be_mode 440 }
end


describe group('root') do
  it { should have_gid 0 }
end

You can add your own tests to the /spec directory.

Create the pipeline

aws cloudformation create-stack --stack-name myteststack --template-body file://pipeline.yaml --parameters ParameterKey=RepositoryName,ParameterValue=<provide repository-name> ParameterKey=ApplicationName,ParameterValue=<provide application-name> ParameterKey=SecurityGroupId,ParameterValue=<provide SecurityGroupId> ParameterKey=VpcId,ParameterValue=<provide VpcId> ParameterKey=SubnetId,ParameterValue=<provide SubnetId> ParameterKey=Region,ParameterValue=<provide Region> ParameterKey=Keypair,ParameterValue=<provide Keypair> ParameterKey=AccountId,ParameterValue=<provide AccountId> --capabilities CAPABILITY_NAMED_IAM

Parameter details

repository-name – The name of the AWS CodeCommit repository

application-name – The Amazon Resource Name (ARNs) are linked with ApplicationName; provide any name

SecurityGroupId – Any security group ID from your AWS account that has port 80 open

VpcId – The ID of your VPC

SubnetId – The ID of a public subnet in your VPC

Region – The AWS Region where you are running this pattern

Keypair – The Secure Shell (SSH) key name to log in to the EC2 instance

AccountId – Your AWS account ID

You can also create a CodePipeline pipeline by using the AWS Management Console and passing the same parameters that are in the previous command line.

Attachments

To access additional content that is associated with this document, unzip the following file: attachment.zip