How to achieve AlwaysOn

When discussing how to achieve High Availability most DBMS focus on handling it via replication. Most of the focus has thus been focused on various replication algorithms.

However truly achieving AlwaysOn availability requires more than just a clever replication algorithm.

RonDB is based on NDB Cluster, NDB has been able to prove in practice that it can deliver capabilities that makes it possible to build systems with less than 30 seconds of downtime per year.

So what is required to achieve this type of availability?

1. Replication
2. Instant Failover
3. Global Replication
4. Failfast Software Architecture
5. Modular Software Architecture
6. Advanced Crash Analysis
7. Managed software

Thus a clever replication algorithm is only 1 of 7 very important parts to achieve the highest possible level of availability. Managed software is one of the addition that RonDB does to NDB Cluster. This won't be discussed in this blog.

Instant Failover means that the cluster must handle failover immediately. This is the reason why RonDB implements a Shared Nothing DBMS architecture. Other HA DBMS such as Oracle and MySQL InnoDB Cluster and Galera Cluster relies on replaying the logs at failover to catch up. Before this catch up has happened the failover hasn't completed. In RonDB every updating transaction updates both data and logs as part of the changing transaction, thus at failover we only need to update the distribution information.

In a DBMS updating information about node state is required to be a transaction itself. This transaction takes less than one millisecond to perform in a cluster. Thus in RonDB the time it takes to failover is dependent on the time it takes to discover that the node has failed. In most cases the reason for the failure is a software failure and this usually leads to dropped network connections which are discovered within microseconds. Thus most failovers are handled within milliseconds and the cluster is repaired and ready to handle all transactions again.

The hardest failure to discover are the silent failures, this can happen e.g. when the power on a server is broken. In this case the time it takes is dependent on the time configured for heartbeat messages. How low this time can be set is dependent on the operating system and how much one can depend on that it sends a message in a highly loaded system. Usually this time is a few seconds.

But even with replication and instant failover we still have to handle failures caused by things like power breaks, thunderstorms and many more problems that cause an entire cluster to fail. A DBMS cluster is usually located within a confined space to achieve low latency on database transactions.

To handle this we need to handle failover from one RonDB cluster to another RonDB cluster. This is achieved in RonDB by using asynchronous replication from one cluster to another. This second RonDB cluster needs to physically separated from the other cluster to ensure higher independence of failures.

Actually having global replication implemented also means that one can handle complex software changes such as if your application does a massive rewrite of the data model in your application.

Ok, are we done now, is this sufficient to get a DBMS cluster which is AlwaysOn.

Nope, more is needed. After implementing these features it is also required to be able to quickly find the bugs and be able to support your customers when they hit issues.

The nice thing with this architecture is that a software failure will most of the time not cause anything more than a few aborted transactions which the application layer should be able to handle.

However in order to build an AlwaysOn architecture one has to be able to quickly get rid of bugs as well.

When NDB Cluster joined MySQL two different software architectures met each other. MySQL was a standalone DBMS, this meant that when it failed the database was no longer available. Thus MySQL strived to avoid crashes since that meant that the customer no longer could access its data.

With NDB Cluster the idea was that there would always be another node available to take over if we fail. Thus NDB, and thus also RonDB implements a Failfast Software Architecture. In RonDB this is implemented using a macro in the RonDB called ndbrequire, this is similar how most software uses assert. However ndbrequire stays in the code also when we run in production code.

Thus every transaction that is performed in RonDB causes thousands error checks to be checked. If one of those ndbrequire's returns false we will immediately fail the node. Thus RonDB will never proceed when we have an indication that we have reached a disallowed state. This ensures that the likelihood of a software failure leading to data being incorrect is minimised.

However crashing solves only the problem as a short-term solution. In order to solve the problem for real we also have to fix the bug. To be able to fix bugs in a complex DBMS requires a modular software architecture. RonDB software architecture is based on experiences from AXE, this is a switch developed in the 1970s at Ericsson.

The predecessor of AXE at Ericsson was AKE, this was the first electronic switch developed at Ericsson. It was built as one big piece of code without clear boundaries between the code parts. When this software reached sizes of millions of lines of code it became very hard to maintain the software.

Thus when AXE was developed in a joint project between Ericsson and Telia (a swedish telco operator) the engineers needed to find a new software architecture that was more modular.

The engineers had lots of experiences of designing hardware as well. In hardware the only path to communicate between two integrated circuits is by using signals on an electrical wire. Since this made it possible to design complex hardware with small amount of failures, the engineers reasoned that this architecture should work as a software architecture as well.

Thus the AXE software architecture used blocks instead of integrated circuits and signals instead of electrical signals. In modern software language these would have been called modules and messages most likely.

A block owns its own data, it cannot peek at other blocks data, the only manner to communicate between blocks is by using signals that send messages from one block to another block.

RonDB is designed like this with 23 blocks that implements different parts of the RonDB software architecture. The method to communicate between blocks is mainly through signals. These blocks are implemented as large C++ classes.

This software architecture leads to a modular architecture that makes it easy to find bugs. If a state is wrong in a block it can either be caused by code in the block, or by a signal sent to the block.

In RonDB signals can be sent between blocks in the same thread, to blocks in another thread in the same node and they can be sent to a thread in another node in the cluster.

In order to be able to find the problem in the software we want access to a number of things. The most important feature to discover is to discover the code path that led to the crash.

In order to find this RonDB software contains a macro called jam (Jump Address Memory). This means that we can track a few thousand of the last jumps before the crash. The code is filled with those jam macros. This is obviously an extra overhead that makes RonDB a bit slower, but to deliver the best availability is even more important than being fast.

Just watch Formula 1, the winner of Formula 1 over a season will never be a car that fails every now and then, the car must be both fast and reliable. Thus in RonDB reliability has priority over speed even though we mainly talk about the performance of RonDB.

Now this isn't enough, the jam only tracks jumps in the software, but it doesn't provide any information about which signals that led to the crash. This is also important. In RonDB each thread will track a few thousand of the last signals executed by the thread before the crash. Each signal will carry a signal id that makes it possible to follow signals being sent also between threads within RonDB.

Let's take an example of how useful this information is. Lately we had an issue in the NDB forum where a user complained that he hadn't been able to produce any backups the last couple of months since one of the nodes in the cluster failed each time the backup was taken.

In the forum the point in the code was described in the error log together with a stack trace of which code we executed while crashing. However this information wasn't sufficient to find the software bug.

I asked for the trace information that includes both the jam's and the signal logs of all the threads in the crashed node.

Using this information one could quickly discover how the fault occurred. It would only happen in high-load situations and required very tricky races to occur, thus the failure wasn't seen by most users. However with the trace information it was fairly straightforward to find what caused the issue and based on this information a work-around to the problem was found as well as a fix of the software bug. The user could again be comfortable by being able to produce backups.

RonDB and Docker Compose

After publishing the Docker container for RonDB I got a suggestion to simplify it further by using Docker Compose. After a quick learning using Google I came up with a Docker Compose configuration file that will start the entire RonDB cluster and stop it using a single command.

First of all I had to consider networking. I decided that using an external network was the best solution. This makes it easy to launch an application that uses RonDB as a back-end database. Thus I presume that an external network has been created with the following command before using Docker Compose to start RonDB:

docker network create mynet --subnet=192.168.0.0/16

The docker-compose.yml is available on GitHub at

https://github.com/logicalclocks/rondb-docker

In the file rondb/21.04/docker-compose.yml for RonDB 21.04 and in rondb/21.10/docker-compose.yml for RonDB 21.10. Link to docker-compose.yml

To start a RonDB cluster now run this command from a directory where you have placed docker-compose.yml.

docker-compose up -d

After about 1 minute the cluster should be up and running and you can access it using:

docker exec -it compose_test_my1_1 mysql -uroot -p

password: password

The MySQL Server is available at port 3306 on IP 192.168.0.10 using the mynet subnet

When you want to stop the RonDB cluster use the command:

docker-compose stop

Docker Compose creates normal Docker containers that can be viewed using docker ps and docker logs commands as usual.

RonDB and Docker

There was a request to be able to test RonDB using Docker. This is now working.

These commands will set up a RonDB cluster on your local machine that can be used to test RonDB:

Step 1: Download the Docker containers for RonDB

docker pull mronstro/rondb

Step 2: Create a Docker subnet

docker network create mynet --subnet=192.168.0.0/16

Step3: Start the RonDB management server

docker run -d \

  --net=mynet \

  -v /path/datadir:/var/lib/rondb \

  -ip 192.168.0.2 \

  -name mgmt1 \

  mronstro/rondb ndb_mgmd --ndb-nodeid=65

Step 4: Start the first RonDB data node

docker run -d \

  --net=mynet \

  -v /path/datadir:/var/lib/rondb \

  -ip 192.168.0.4 \

  -name ndbd1 \

  mronstro/rondb ndbmtd --ndb-nodeid=1

Step 5: Start the second RonDB data node

docker run -d \

  --net=mynet \

  -v /path/datadir:/var/lib/rondb \

  -ip 192.168.0.5 \

  -name ndbd2 \

  mronstro/rondb ndbmtd --ndb-nodeid=2

Step 6: Check that the cluster has started and is working

This step isn't required, but just to show that the cluster is

up and running, start the RonDB management client and issue the

show command.

docker exec -it mgmt1 ndb_mgm

ndb_mgm> show

This should hopefully show a starting cluster and after about

half a minute the cluster should be started.

Step 7: Start a MySQL Server

Note that the MySQL Server uses /var/lib/mysql as datadir internally

whereas the RonDB management server and data node uses

/var/lib/rondb.

docker run -d \

  --net=mynet \

  -v /path/datadir:/var/lib/mysql \

  -e MYSQL_ROOT_PASSWORD=your_password \

  -ip 192.168.0.10 \

  -name mysqld1 \

  mronstro/rondb mysqld --ndb-cluster-connection-pool-nodeids=67

Step 8: Start a MySQL client

docker exec -it mysqld1 mysql -uroot -p

Password: your_password

Now you are connected to a MySQL client that can issue SQL commands

towards the RonDB cluster. Below is a very simple example of such

commands:

mysql> CREATE DATABASE TEST;

mysql> USE TEST;

mysql> CREATE TABLE t1 (a int primary key) engine=ndb;

mysql> INSERT INTO t1 VALUES (1),(2);

mysql> SELECT * FROM t1;

I tested this on my development machine using Mac OS X. To succeed with the setup

my Docker setup required at least 8 GByte of memory. RonDB is optimised for use

in VMs in the cloud where a minimum of 8 GByte of memory is available for the

data node VMs. Since the default configuration of Docker will presumably mainly

be used for simple tests I decided to decrease the size of the RonDB data nodes

such that they fit in 3 GBytes of memory. It is definitely possible to run

RonDB in an even smaller environment, but I think that the default should at least

be able to load at least 1 GByte of data and a fair amount of tables into RonDB.

RonDB and Docker is documented at https://docs.rondb.com/rondb_docker/

The RonDB documentation has also been improved at the same time.

The GitHub tree for the Docker containers can be found at:

https://github.com/logicalclocks/rondb-docker

The GitHub tree is based on the MySQL Docker tree at GitHub.

The Docker Hub is found at:

https://hub.docker.com/r/mronstro/rondb/