In this post I’m going to explore Simple Workflow Service (SWF) available in AWS.

To understand what SWF is good for, we need to first understand what a workflow is. Wikipedia defines it as follows:

A workflow consists of an orchestrated and repeatable pattern of activity, enabled by the systematic organization of resources into processes that transform materials, provide services, or process information. It can be depicted as a sequence of operations, the work of a person or group, the work of an organization of staff, or one or more simple or complex mechanisms.

In computer systems we care about the part about processing information. Some things that could be modeled as workflows:

• Deployment pipeline: We could receive some code as input and then build it in a worker machine. We can run tests in parallel in different machines. If all tests pass we can deploy the binaries to another set of machines.
• Coordinate shipments: A user buys a product on an online store and the order is placed on a system. A human monitors this system and takes care of finding the products in a warehouse and shipping them to the correct address. When the shipment is made, the information is entered in a system. The workflow notices this information an e-mails the user the shipping details.
• Asynchronous image processing: A system uploads files to a system for processing (let’s say, create thumbnails). A workflow uses multiple workers to execute the task. If any of the machines fails while processing a set of files, they same work can be taken over by another worker.

In the beginning of times, there were centralized version control systems (SVN and Perforce are examples). This means that there is a server somewhere that contains all our code and the history of all the changes. If someone needs to work on that codebase they do a checkout (typically of the main branch) and they will get the newest version of all the files.

If the server looks something like this (Every letter represents a different commit):

When a developer checks out main, they will get only the files at D, the commit history exists only in the server.

This has a two main disadvantages:

• It is not possible to create local branches. If a developer needs a branch they have to push it to the server
• If the server explodes, all the history is lost

CloudFormation is AWS’ offering for modeling infrastructure as code. Its purpose is similar to that of Salt or Terraform.

Getting started

CloudFormation allows us to define our infrastructure on template files written in JSON or YAML. The following examples show a template to create an EC2 instance:

1
2
3
4
5
6
7
8
9
10
11
12
{
  "Description": "Create a single EC2 instance",
  "Resources": {
    "Host1": {
      "Type" : "AWS::EC2::Instance",
      "Properties": {
        "InstanceType": "t2.micro",
        "ImageId": "ami-003634241a8fcdec0"
      }
    }
  }
}

Have you ever had to draw an architecture diagram and found the repetitive clicking and dragging tedious? Did you have to do modifications to that diagram and found it complicated?

Graphviz is an open source graph visualization software that allows us to decribe a diagram using code, and have it automatically drawn for us. If the diagram needs to be modified in the future, we just need to modify the description and the nodes and edges will be repositioned automatically for us.

Drawing graphs

Before we start writing graphs, we need to learn how we can convert our code into an image so we can test what we are doing.

In my path to learning about networking on AWS I have written a few articles:

• Introduction to AWS networking
• Setting up a bastion host on AWS
• Introduction to AWS NAT Gateway

This time I’m going to write about a way to allow a private EC2 instance to communicate with an AWS service without having to go through the public Internet. At the time of this writing, there are two services that provide VPC Gateway endpoints: S3 and DynamoDB.

We might want to use a VPC Gateway endpoint to improve security and decrease latency when a service we own needs to use S3 or DynamoDB. Without VPC Gateway endpoints, we would have our private instance use a NAT Gateway to reach the Internet (Including any AWS service). With a VPC Gateway endpoint the traffic stays inside AWS network, making it faster and safer.

A NAT (Network Address Translation) Gateway can be used to allow an instance in a private Subnet to communicate with the Internet while preventing the Internet from initiating connections to it.

In my previous article I explained how to create a bastion host. In this article I’m going to create a private Subnet in the same VPC, and I’m going to allow this Subnet to initiate connections to the Internet without giving the instances a public IP address.

The end result of my article about creating a bastion host was this:

If you are not familiar with networking concepts on AWS, I recommend you take a look at my introduction to aws networking.

A Bastion host (also called Jumpbox) is used to protect hosts that are part of a private network, while still allowing access to them over the Internet. If a system administrator needs to access other hosts, It needs to first SSH to the Bastion and from there, SSH to any other host.

Being exposed to the Internet, the Bastion becomes the target of attackers and should be a central part of our security plan.

A few months ago, I wrote an introduction to networking for Google Cloud. Today I find myself working with AWS, so I’m going to explore networking on the AWS platform.

I’m going to be using AWS CLI for my examples, so I recommend you install it and configure it before proceeding.

Virtual Private Clouds (VPC), Subnets and Security Groups (SG)

To get started we need to get familiar with these 3 fundamental concepts:

• Virtual Private Cloud (VPC) - Refers to a network that is logically isolated from the rest of the world. A VPC is a regional resource (It can span a full region, but not accross regions)
• Subnet - A section of a VPC. Subnets exist in a single Availability Zone (AZ)
• Security Group (SG) - A virtual Firewall. Any EC2 instance must be attached to at least one Security Group. By default a Security Group allows all outbound traffic and disallow all inbound traffic

In a previous post I wrote about AWS CodeBuild, which allows us to run our builds using AWS infrastructure. In this post we are going one step further and explore CodePipeline; AWS’ solution for continuos delivery.

Some of Pipelines’ features:

• Detect code changes and start Pipeline automatically
• Split releases into stages (One per environment, for example)
• Pause the releases if a step fails
• Allow steps to only proceed after manual approval

CodeBuild is AWS’ offering for running builds in the cloud. We can think of it as an alternative to TravisCI or CircleCI.

Concepts

There are 4 things we need to configure as part of a CodeBuild project:

• Source - Get the code we want to build. At the time of this writing, we can retrieve code from S3, GitHub, Bitbucket or CodeCommit (AWS’ code hosting offering)
• Environment - Type of machine to use for the builds
• Buildspec - Commands to run as part of the build
• Artifacts - Artifacts to publish to S3 (Optional)