Category Archives: Linux

Getting familiar with Terraform

In a previous post I covered the basics of Terraform. In this post I’m going to cover a few more things that I find necessary in most infrastructures I create.

The machines

I’m going to start with a couple of machines:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// Configure Google Cloud
provider "google" {
  credentials = "${file("credentials.json")}"
  project = "ncona-1504"
  version = "~> 1.13"
}

// Machines
resource "google_compute_instance" "us-central1-c--f1-micro--001" {
  name         = "us-central1-c--f1-micro--001"
  machine_type = "f1-micro"
  zone         = "us-central1-c"

  boot_disk {
    initialize_params {
      image = "ubuntu-1604-xenial-v20170815a"
    }
  }

  network_interface {
    network = "default"
  }
}

resource "google_compute_instance" "us-central1-c--f1-micro--002" {
  name         = "us-central1-c--f1-micro--002"
  machine_type = "f1-micro"
  zone         = "us-central1-c"

  boot_disk {
    initialize_params {
      image = "ubuntu-1604-xenial-v20170815a"
    }
  }

  network_interface {
    network = "default"
  }
}

Read more »

Taking over existing instances with Terraform

A few days ago, while playing with Terraform I realized that I want Terraform to manage some instances that I had already created in Google Cloud. Because these instances existed before I was using Terraform, it doesn’t konw anything about them.

The first thing I had to do to take over them, was to add them to the configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
resource "google_compute_instance" "us-central1-c--f1-micro--001" {
  name         = "us-central1-c--f1-micro--001"
  machine_type = "f1-micro"
  zone         = "us-central1-c"

  boot_disk {
    initialize_params {
      image = "ubuntu-1604-xenial-v20170815a"
    }
  }

  network_interface {
    network = "default"
  }
}

Once I had the machine in my configuration, I just had to tell Terraform, which machine is that:

1
terraform import google_compute_instance.us-central1-c--f1-micro--001 ncona-179804/us-central1-c/us-central1-c--f1-micro--001

From here terraform can manage it as if it had created it.

Terraform

Terraform is a tool for managing infrastructure. It works with most cloud providers out there, but it can also (although with considerably more work) integrate with any custom solutions a company might have.

Managing infrastructure

What does it mean to manage infrastructure? Although different infrastructure management systems have different features, there are a few features that are present in most systems:

  • Asset lifecycle – Adding or removing assets (machines, load balancers, switches, etc…)
  • Networking – Managing how the assets communicate (networks, routes, permissions, etc…)
  • Security – Secure the network and machines so it can only be accesses by the intended people
  • Updates – Update the system software when necessary for security or stability reasons

Why Terraform?

There are many tools out there that allow you to manage your infrastructure (Chef, Ansible, Salt, etc…). I have worked in companies that run successfully using Chef, Ansible and Salt, so I’m sure any of the tools can be made to work. I’m writing this post about Terraform because I heard good things about it from people I admire. Two things that usually come up are an easy way to manage your whole infrastructure using configuration files, as well as making optimal changes that can be applied quickly without affecting dependent services.

Read more »

Linux page cache

Linux (and most other operating systems) provide a transparent layer of caching for auxiliary storage (hard drives, etc…). This layer allows fast access to frequently used files on disk by keeping their content in memory and reading from there when necessary.

The kernel can use any free space in RAM as page cache. If the system requires more memory, the kernel might free space used by this cache and provide it to the application that needs it.

Inspecting page cache

Page cache is stored in RAM, but the space can be reclaimed by the kernel for applications whenever necessary. In Linux we can see how many bytes of RAM are used for page cache:

1
2
3
4
~ $ free -h
              total        used        free      shared  buff/cache   available
Mem:            19G        5.4G        7.8G        1.5G        6.2G         12G
Swap:           19G          0B         19G

Read more »

File descriptors

I’ve heard a few times during my career references to file descriptors without knowing much about them. Today I’m going to try to understand what they are.

File descriptors are a Unix concept. They refer to a way of referencing a file or other resources (stdin, stdout, etc…). They are referenced by unsigned integers. Some old versions of Unix used to limit the number of file descriptors per process to 20 (0 to 19), but nowadays, there is no actual limit on the number of file descriptors a process can have.

Each running process is assigned a file descriptors table. The table for a process can be found at /proc/PID/fd. We can see the file descriptors used by a process like this:

Read more »

Introduction to etcd

In previous posts I wrote a little about distributed systems and the Raft algorithm. Today I’m going to look at one distributed key-value store that uses the Raft algorithm to achieve consistency and high availability.

From a client’s perspective, etcd will behave like any other key value store out there. It’s use of Raft underneath will make sure that there is only one leader at a given time and that the log is replicated to all nodes.

Getting ready

For this exercise I’m going to create a 5-node cluster, but before we start there are a few things we need to decide.

By default each etcd nodes uses port 2380 for communicating with clients and port 2379 for server to server communication. We will keep this default behavior.

Each node in the cluster needs to be able to communicate with the rest of the nodes in the cluster. The number of nodes in the cluster and their location needs to be configured for the cluster to be able to do some work.

In normal conditions we would have each node in a different host with a different IP Address. This would allow us to say something like: You can find node A at 10.10.10.2.

Running the cluster in a single machine makes things challenging because they would all be sharing the same IP address. To walk around this issue, we will create our own docker network and work within this network.

Read more »

Playing with Kubernetes locally with Minikube

I want to start using Kubernetes to manage my services. Before I go all-in I will do some playing around with their easy-to-install version that can run on a single machine.

Requirements

Before we can run Minikube locally we need to have a virtualization solution installed in our host. Since my computer supports KVM, I decided to go with it. You can check if your computer supports KVM with this command:

1
egrep -c '(vmx|svm)' /proc/cpuinfo

Read more »

Modify the name of the Desktop and Downloads folders on Ubuntu

To modify the names of your Desktop and Downloads folder you just modify ~/.config/user-dirs.dirs:

1
2
XDG_DESKTOP_DIR="$HOME/desktop"
XDG_DOWNLOAD_DIR="$HOME/downloads"

Disable SSLv3 in HAProxy

I just learned that my load balancer is vulnerable to the POODLE attack due to SSL 3. The recommended solution is to disable SSL 3.

I explained my HAProxy setup in a previous post, and also how I do SSL termination.

The section from my configuration I care about is:

1
2
3
4
5
6
frontend https-in
        bind *:443 ssl crt /certs/ncona.pem

        acl ncona-web-frontend hdr(host) -i ncona.com www.ncona.com

        use_backend ncona-web if ncona-web-frontend

This mode is called SSL offloading in HAProxy terms. Fixing it is as simple as adding a keyword (no-sslv3):

1
        bind *:443 ssl crt /certs/ncona.pem no-sslv3

Getting Rails to run in an Alpine container

I’m trying to get a little Rails application ready for production, and just for fun I’m trying to make the image a little slimmer than it currently is (860 MB).

There is a official docker image of ruby with alpine (ruby:alpine), but because of the libraries that rails uses (with native bindings), it is a little more challenging than just referencing that image on the top of the Dockerfile.

Solving the issues

I added FROM ruby:2.4.1-alpine at the top of my Dockerfile and tried to create the image. The first problem I faced was with myql2. For the mysql2 gem to work on Alpine it is necessary to have a compiler (build-base) and MySQL development libraries (mariadb-dev). I added this to my Dockerfile:

Read more »