In previous posts I wrote a little about distributed systems and the Raft algorithm. Today I’m going to look at one distributed key-value store that uses the Raft algorithm to achieve consistency and high availability.

From a client’s perspective, etcd will behave like any other key value store out there. It’s use of Raft underneath will make sure that there is only one leader at a given time and that the log is replicated to all nodes.

Getting ready

For this exercise I’m going to create a 5-node cluster, but before we start there are a few things we need to decide.

By default each etcd nodes uses port 2380 for communicating with clients and port 2379 for server to server communication. We will keep this default behavior.

Each node in the cluster needs to be able to communicate with the rest of the nodes in the cluster. The number of nodes in the cluster and their location needs to be configured for the cluster to be able to do some work.

In normal conditions we would have each node in a different host with a different IP Address. This would allow us to say something like: You can find node A at 10.10.10.2.

Running the cluster in a single machine makes things challenging because they would all be sharing the same IP address. To walk around this issue, we will create our own docker network and work within this network.

In a past post I wrote about distributed systems, but I intentionally omitted the subject of leaderless replication since I consider it a topic that deserves a little more focus.

In this post I’m going to explore how a leaderless system works and explain a little about how the Raft algorithm helps us achieve consensus.

Leaderless replication

As the name suggests, there are no leaders (or masters) in a leaderless setup. In this configuration all instances can receive reads and writes. If you read my previous post, you might be thinking that this sounds like master-master replication, but it is actually very different.

I mentioned two main problems in a master-master setup: Replication lag when you write to one master and then read from the other, and conflicts when you modify the same record in both masters and they try to sync. Leaderless replication doesn’t have these problems (it has others that I’ll explore soon). On top of not having those problems, a leaderless system can stay up even when instances are down (like the master-master configuration).

In computer science a distributed system is a software system in which different parts of it communicate by passing messages through a network. The different parts could be running in the same machine or distributed across the globe; as long as they communicate through an unreliable channel (a network), we can classify them as distributed and consider the challenges that come with it.

With this definition in mind, we could think about many examples of distributed systems. A single monolithic application communicating with a database could be considered a distributed system if the application communicates through a network protocol.

Although in practice, local networks can be pretty reliable, they are still vulnerable. There are two condition that can cause a plethora of problems to a system: the network being down or the network being slow. These two conditions can put the system in a wide variety of states that may give results we don’t expect.

Before we look at how these problems can affect a distributed system, lets look at a distributed system whose failure mode is mostly understood and accepted to this date: a stateless system.

I want to start using Kubernetes to manage my services. Before I go all-in I will do some playing around with their easy-to-install version that can run on a single machine.

Requirements

Before we can run Minikube locally we need to have a virtualization solution installed in our host. Since my computer supports KVM, I decided to go with it. You can check if your computer supports KVM with this command:

1
egrep -c '(vmx|svm)' /proc/cpuinfo

In a previous post I talked a little about creating installable web apps. One important thing that I didn’t cover is how to automatically prompt a user on your web app to add it to their home screen.

Serve over https

One of the requirements for chrome to show an app install banner is that the app is served over HTTPS. This can be done in a few ways depending on your server. I wrote a post on how to get free SSL certificates using Let’s encrypt that should help you get started.

To modify the names of your Desktop and Downloads folder you just modify ~/.config/user-dirs.dirs:

1
2
XDG_DESKTOP_DIR="$HOME/desktop"
XDG_DOWNLOAD_DIR="$HOME/downloads"

I just learned that my load balancer is vulnerable to the POODLE attack due to SSL 3. The recommended solution is to disable SSL 3.

I explained my HAProxy setup in a previous post, and also how I do SSL termination.

The section from my configuration I care about is:

1
2
3
4
5
6
frontend https-in
        bind *:443 ssl crt /certs/ncona.pem

        acl ncona-web-frontend hdr(host) -i ncona.com www.ncona.com

        use_backend ncona-web if ncona-web-frontend

This mode is called SSL offloading in HAProxy terms. Fixing it is as simple as adding a keyword (no-sslv3):

1
        bind *:443 ssl crt /certs/ncona.pem no-sslv3

As I move forward with my Beego project, I have reached a point where I need to authenticate my users. In the flow that I’m looking for, the client (web app, mobile app, etc…), will communicate directly with the Auth provider (Google, Facebook, etc…) and get a JWT. The only thing the server needs to do is validate the JWT. If the validation succeeds, it means the user is logged in.

If you are not familiar with JWT, you can read my previous article that explains how JWT works.

Authentication flow

Since the authentication with Google is going to happen entirely on the client, the server logic becomes a lot simpler. For my application, all endpoints will require the user to be logged in, so I will create a middleware to verify this. The middleware will expect a valid JWT in the Authorization header. If this requirement is not met, the server will return a 401.

Before we start creating models, we need to make sure our app can connect to our database. At the time of this writing Beego supports MySQL, PostgreSQL and Sqlite3.

We can have the ORM connect to our database from our main.go file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
package main

import (
    "os"
    "strings"

    _ "app/routers"

    "github.com/astaxie/beego"
    "github.com/astaxie/beego/orm"
    _ "github.com/go-sql-driver/mysql"
)

func init() {
    orm.RegisterDriver("mysql", orm.DRMySQL)
    parts := []string{os.Getenv("MYSQL_USER"), ":", os.Getenv("MYSQL_PASSWORD"),
            "@tcp(", os.Getenv("MYSQL_HOST"), ":3306)/", os.Getenv("MYSQL_DATABASE")}
    orm.RegisterDataBase("default", "mysql", strings.Join(parts, ""))
}

func main() {
    if beego.BConfig.RunMode == "dev" {
        beego.BConfig.WebConfig.DirectoryIndex = true
        beego.BConfig.WebConfig.StaticDir["/swagger"] = "swagger"
    }
    beego.Run()
}

A few weeks ago I wrote an article showing how to start a simple Beego server. Today I’m going to go a little further and explain how to start building your database using migrations. For my examples I will use a MySQL server with an empty database (no tables) named beego.

It is possible to just log in to a database server and type the SQL commands necessary to create or modify tables when we need to, but using migrations helps keep track of the changes made to the database over time. This provides something similar to version control at the database level.

If we need to create a new table, we would start by creating a new migration file using the bee tool:

1
bee generate migration create_user_table