Using Chef-solo with CloudFormation

1 Introduction

1.1 Traditional Chef environment

Chef is a configuration management tool that allow us to provision software or OS setup on a server with configuration instructions described in a Chef recipe. And collections of recipes are stored in a cookbook.

In a traditional Chef environment setup, we need to utilize a Chef server to stores the cookbooks. And when a new chef client node is added to the Chef server, recipes are sent to the client node and tell it on how to configure itself.

By default, the client node will check in every 30 minutes to see if any chef cookbook has been changed. So by changing the Chef cookbook recipes, the changes will be rolled out over your entire infrastructure automatically.

Following is a diagram on how a traditional Chef environment that requires Chef server and Chef workstation.

1.2 Using Chef without Chef server

The biggest problem on using Chef server is we can not easily integrate Chef server and CloudFormation. In AWS environment, we just want to provision applications on the EC2 instance based on application tag we defined in the CloudFormation template.

To get around the problem of using Chef server, we are using Chef-solo.

Chef-solo is an open source version of the chef-client that do not need access to a Chef server. Chef-solo runs locally on the EC2 instance with the cookbooks, data_bags, roles and environments files stored on the local disk. To configure the application on the EC2 instance, we just need to pull the proper Chef cookbooks from the Git repository based on the application tag value defined with the EC2 instance.

1.3 CodeCommit

AWS CodeCommit is an AWS managed service that allow us to store anything, like source code, documents or binary files. There is no limit on the size of the repository. CodeCommit is also integrated with AWS IAM, data is encrypted at rest.

CodeCommit is also Git compatible, so the learning curve to use CodeCommit as repository is low for our system administrators.

The following diagram shows how AWS CodeCommit is implemented by AWS.

AWS CodeCommit stores data in Amazon Simple Storage Service and Amazon DynamoDB. CodeCommit automatically encrypts files and repositories at rest through AWS Key Management Service.

2 Using Chef-solo with CloudFormation

We will use CloudFormation to provision EC2 instances and then use the CloudFormation helper scripts to install Chef client and then run Chef-solo to configure the application.

The following illustrate the overall process.

The following are the process:  

1. CloudFormation helper script execute a bash command to Install chef client
2. CloudFormation helper script Pull the chef-local script and related configuration files from S3 bucket and store it on the local disk of the EC2 instnace.
3. CloudFormation help script execute the Chef-local.sch
4. Pull the proper chef cookbooks based on the application role tag
5. Run chef client in local mode to configure the application