Tips and tricks for rapid cloudformation template development

1 CloudFormation Development

The CloudFormation templates are either JSON or YMAL data files. It is very common to see templates has hundreds or even thousand of line lines. When these these template grow big, they are hard to understand and maintain. To make matters worse is when you perform

a Stack creation/update, a failure in any single Resource will cause the Stack to rollback the entire set of Resource changes, which can unnecessarily destroy other successfully created Resources and take a very long time.

To solve the problem, we need to break the template development in modules and in small pieces.

1.1 Use a Layered and service-oriented architecture

For deploying Spring Boot applications, we organize our templates is using the combination of layered and service-oriented architecture

We can separate the templates for each environment

In each environment, we separate the templates by service

1.2 Use nested CloudFormation stacks

Breaking down the templates by service helps but we still found out some template grow over a thousand lines, it is really crazy.

To get around the problem, we can use nested CloudFormation stack.

A CloudFormation nested stack is a stack containing one or more CloudFormation stacks as resources. Nesting your stacks allows you to break up your CloudFormation into logical pieces. It also allows you to map outputs to parameters between templates.

2 CloudFormation Deployment

2.1 Using Cross Stack References with Service Catalog

By using ServiceCatalog, we can distribute a tool to end users by importing CloudFormation templates. The service catalog will prompt the use to parameters need in the templates for the resources and configurations

Because the need to enter the parameter manually, it could be error prone. To lesser the chance for error, we can use the Cross Stack References feature in CloudFormation. With this feature we you can export values in one stack and import them in another stack.

2.2 Using Code Pipeline for infrastructure deployment

Normally, people uses CodePipeline to distribute software update to the applications. But one thing no many people aware is we can use CodePipeline to create infrastructure and propagate changes on the Cloudformation template from one environment to another

The following diagram illustrate the process on how to use CodePipeline to push the changes on environment.

                       

3 Debugging

3.1 JSON file editor

You need an editor that can suggests typos with color highlighting. I use Atom on my Mac OS and it works well.

3.2 Syntax testing

You can use Amazon CLI tool to check syntax error, but I prefer to use CloudFormation Designer to check  syntax error, just go to AWS console -> cloudformation -> Design template. Simply paste your template (JSON) on the "Template" pane and then click on the tick symbol to validate your template. Any errors will show up in the "Error" pane.

3.3 Prevent type mismatch in nested CloudFormation template

A recent new feature added to Cloudformation this past December was the addition of additional Parameter Types. These new Types allow your templates to perform stronger data checking, and also can "fail-fast" when creating resources and nested Cloudformation stacks. You also have the ability to provide nicer human-readable custom error messages when invalid values are passed in using the new ConstraintDescription attribute.

The new types are especially helpful when dealing with various VPC resources. You can ensure that Parameters for your templates are the correct type, and are explicit about expecting a single value vs. a List.

For example:

"Parameters" : { "SingleGroup": { "Type": "AWS::EC2::SecurityGroup::Id", ...}, "GroupList": {"Type": "List<AWS::EC2::SecurityGroup::Id>", ...} }

3.4 Log file for Cloud-init and Cfn-init

For provision EC2 instance with application, we need to bootstrap the instance with the “user-data” (or cloud-init) and CloudFormation help scripts (cfn-init).  We can view the logs, such as /var/log/cloud-init.log or /var/log/cfn-init.log , to help use to the debug the bootstraping process.

4 Sample templates structure

Following is an example of the CloudFormation templates to deploy an entire environment.

• Network layer
• VPC.template creates:
• VPC
• DCHP option set
• Internet Gateway
• Subnets
• Network ACL
• NAT gateway
• route tables
• SecurityGroups.template create:
• ELB security groups
• Application server security groups
• ELBs.template creates:
• Elastic load balancers
• Roles.template creates
• Application server roles
• BGP VPN server role
• Rundeck access role
• Application layer
• EC2instance.template can be used to create EC2 instance with a application tag to tell which Chef cookbook to apply to the EC2 instance.

5 Reference

• Understanding Nested CloudFormation Stacks
• Cloud and DevOps ConsultantsCloudFormation Zen