Friday, November 15, 2019

What's new in MySQL Cluster 8.0.18

MySQL Cluster 8.0.18 RC2 was released a few weeks back packed with a set of
new interesting things.

One major change we have done is to increase the number of data nodes from 48 to
144. There is also ongoing work to fully support 3 and 4 replicas in MySQL
Cluster 8.0. NDB has actually always been designed to handle up to 4 replicas.
But the test focus have previously been completely focused on 2 replicas. Now we
expanded our test focus to also verify that 3 and 4 replicas work well. This means
that with NDB 8.0 we will be able to confidently support 3 and 4 replicas.
This means that with NDB 8.0 it will be possible to have 48 node
groups with 3 replicas in each node group in one cluster.

The higher number of nodes in a cluster gives the possibility to store even more
data in the cluster. So with 48 node groups it is possible to store 48 TByte of
in-memory data in one NDB Cluster and on top of that one can also have
about 10x more data in disk data columns. Actually we have successfully
managed to load 5 TByte of data into a single node using the DBT2 benchmark,
so with 8.0 we will be able to store a few hundred TBytes of replicated
in-memory and petabytes of data in disk data columns for key-value stores
with high demands on storage space.

Given that we now support so much bigger data sets it is natural that we focus
on the ability to load data at high rates, both into in-memory data and into
disk data columns. For in-memory data this was solved already in 7.6 and
there is even more work in this area ongoing in 8.0.

We also upped one limitation in NDB where 7.6 have a limitation on row sizes
up to 14000 bytes, with NDB 8.0 we can handle 30000 byte row sizes.

Another obvious fact is that with so much data in the cluster it is important to
be able to analyze the data as well. Already in MySQL Cluster 7.2 we
implemented a parallel join operator inside of NDB Cluster available
from the MySQL Server for NDB tables. We made several important
improvements of this in 7.6 and even more has happened in NDB 8.0.

This graph shows the improvement made to TPC-H queries in 7.6 and in
8.0 up until 8.0.18. So chances are good that you will find that some of
your queries will be executed substantially faster in NDB 8.0 compared to
earlier versions. NDB is by design a parallel database machine, so what
we are doing here is ensuring that this parallel database machine can now
also be applied for more real-time analytics. We currently have parallel
filtering, parallel projection and parallel join in the data nodes. With
NDB 8.0 we also get all the new features of MySQL 8.0 that provides a
set of new features in the query processing area.
 The main feature added in 8.0.18 for query processing is the ability to pushdown
join execution of queries where we have conditions of the type t1.a = t2.b.
Previously this was only possible for the columns handled by the choosen index
in the join. Now it can be handled for any condition in the query.

8.0.18 also introduces a first step of improved memory management where the goal
is to make more efficient use of the memory available to NDB data nodes and also
to make configuration a lot simpler.

In NDB 8.0 we have also introduced a parallel backup feature. This means that taking
a backup will be much faster than previously and load will be shared on all LDM threads.

Wednesday, October 23, 2019

Setting up an NDB Cluster in the Oracle Cloud using Auto Installer

In MySQL Cluster 8.0.18 we have developed MySQL Cluster Auto Installer to
also support installing NDB :)

We have made it very easy to setup an NDB Cluster in the Oracle Cloud.
The Auto Installer will take care of installing the proper software, installing
firewalls, installing some supportive software. Most of the testing of this
software have been done against Oracle Cloud using instances with Oracle
Linux 7.

I prepared two Youtube videos to show how it works. The first one gives some
insights into setting up the Compute Instances required in the Oracle Cloud.

Setup compute instances in Oracle Cloud for MySQL Cluster AutoInstaller

The second video uses these compute instances to set up an NDB Cluster.

Setting up an NDB Cluster in the Oracle Cloud using Auto Installer

Tuesday, October 22, 2019

Setting up MySQL Cluster on local machine using AutoInstaller

We have a new version of the MySQL Cluster Auto Installer. I have prepared 3
Youtube videos that shows how to make use of the Auto Installer to install and
set up a cluster on your local machine.

This is my first attempt at making Youtube videos to explain things around
MySQL Cluster.

The Auto Installer is intended as a tool to make it easy to get a cluster up and
running. It is NOT intended for managing a cluster.

The first Youtube video Install MySQL Cluster 8.0.18 on Mac OS X shows how
to install MySQL Cluster 8.0.18 on Mac OS X. This is obviously trivial, so should
be straightforward to do even without the video.

The second Youtube video Starting a local MySQL Cluster using Auto Installer sets
up a small cluster with 1 management server, 2 data nodes and 1 MySQL Server and
explains some details around this.

Using MySQL Cluster AutoInstaller to start development with NDB goes a bit deeper
and also shows how to make use of the cluster and do some trivial operations
through a MySQL client and the NDB management client. It also shows how one can
extend the possible configurations supported by directly manipulating configuration
files deployed by the Auto Installer.

A little note for Mac OS X users with high resolution is that full 4K resolution is
available through Google Chrome, not through Safari.

Friday, April 05, 2019

Manual for dbt2-0.37.50.15, fully automated Sysbench and DBT2 benchmarking with NDB

The link dbt2.0.37.50 manual provides the details of how to use the dbt2-0.37.50 scripts
to execute benchmarks using MySQL Cluster.

These scripts can be used to execute automated test runs of Sysbench, DBT2 and
FlexAsynch. I also use it to start up NDB Clusters to run DBT3 benchmarks and
YCSB benchmarks.

This set of scripts originates from 2006 when I wanted to automate all my benchmark
efforts. The most challenging benchmarks constitute starting more than 100 programs
to work together and using more than 100 machines. This requires automation to
be succesful.

Now running any benchmark is a 1-liner e.g.
./bench_run.sh --default-directory /path/to/dir --init

The preparation to run this benchmark is to place a file called autobench.conf in
/path/to/dir. This contains the configuration of the NDB data nodes, NDB MGM
servers, MySQL Servers and the benchmark programs. Multiple benchmark
programs are supported for Sysbench, DBT2 and flexAsynch.