Showing posts with label ARM64. Show all posts
Showing posts with label ARM64. Show all posts

Thursday, March 23, 2023

Laptop vs Desktop for RonDB development

 Most developers today use laptops for all development work. For the last 15 years I have considered desktops and laptops to be very similar in performance and use cases. This is no longer the case as I will discuss in this blog.

Personally I use a mix of laptops and desktops. For me the most important thing as a developer is the screen resolution and the speed of compilation. But I have now found that desktops can be very useful for the test phase of a development project, in particular the later testing phases.

Many years ago I read that one can increase productivity by 30% by using a screen with higher resolution thus fitting more things at the same time on the screen. Personally I have so far found 27" screens to be the best size, larger size means neck pain and smaller means that productivity suffers. The screen resolution should be as high as your budget allows.

My experience is that modern laptops can be quite efficient in compilation. There is very little difference in compilation time towards desktops.

However recently I tested running our new RonDB Docker clusters on laptops and desktops. What I have seen is that the performance of these tests can differ up to 4x.

I think the reason for this large difference is that desktops can sustain high performance for a long time. Some modern desktops can handle CPUs that use more than 200W whereas most laptops will be limited to about 45W. For a compilation that only runs for about 5 minutes and have some serialisation the difference becomes very small. The most important part for compilation is how fast the CPU is on single-threaded performance and that it can scale the compilation to a decent number of CPUs.

However running a development environment for RonDB means running a cluster on a single machine where there are two data node processes, two MySQL server processes and a management process and of course any number of application processes. A laptop can handle this nicely and the performance for a single-threaded application is the same for laptop and desktop. However when scaling the test to many threads the laptop hits a wall whereas the desktop simply continues to scale.

The reason is twofold, the desktop CPUs can have more CPU cores. Most high-end laptops today have around 8-10 CPU cores. The high-end desktops today however goes to around 16-24 CPU cores. In addition the desktop can usually handle more than 4x as much power. The power difference and the core difference delivers a 4x higher throughput in heavy testing.

Thus my conclusion is that laptops are good enough for the development phase together with an external screen. However when you enter the testing phase when you need to run heavy functional tests and load tests on your software a desktop or a workstation will be very useful.

In my tests on a high-end desktop I ran a Sysbench OLTP RW benchmark using the RonDB Docker environment, I managed to run up to 15.000 TPS. This means running 300.000 SQL queries per second towards the MySQL servers and the data nodes. The laptop could handle roughly 25% of this throughput.

Obviously the desktop could be a virtual desktop in the modern development environment. But a physical machine is still a lot more fun.

RonDB is part of the Hopsworks Feature Store platform.


Monday, January 31, 2022

RonDB receives ARM64 support and large transaction support

 RonDB is the base platform for all applications in Hopsworks. Hopsworks is a machine learning platform featuring a Feature Store that can be used in online applications as well as offline applications.

This means that RonDB development is driven towards ensuring that operating RonDB in this environment is the best possible.

RonDB is designed for millions of small transactions reading and writing data. However occasionally applications perform rather large transactions. Previous versions of RonDB had some weaknesses in this area. The new versions of RonDB now supports also large transactions although the focus is still on many smaller transactions.

Designing this new support of large transactions required a fairly large development effort. To do this in a stable release is a challenge, therefore it was decided to combine this effort with a heavy testing period focused on fixing bugs.

This effort has been focused on achieving three objectives. First to stabilise the new RonDB 21.04 releases which is the stable release of RonDB. Second, to stabilise the next RonDB release at the same level as RonDB 21.04. Third, we also wanted the same level of support for ARM64 machines.

We are now proud to release RonDB 21.04.3, a new stable release of RonDB that supports much larger transactions. Since the release of RonDB 21.04.1 in July 2021 we have fixed more than 50 bugs in RonDB and we are very satisfied with the stability also on ARM64 machines.

The original plan was to release the next version of RonDB in October 2021, however we didn't want to release a new version with any less stability than the RonDB 21.04 release. Thus instead we release this new version of RonDB now, RonDB 22.01.0.

ARM64 support covers both RonDB 21.04.3 and RonDB 22.01.0. RonDB is now also supported on both Linux and Mac OS X and on Windows it is supported using WSL 2 (Linux on Windows) on Windows 11. We have extensively tested RonDB on the following platforms:

  1. Mac OS X 11.6 x86_64
  2. Mac OS X 12.2 ARM64
  3. Windows WSL 2 Ubuntu x86_64
  4. Ubuntu 21.04 x86_64
  5. Oracle Linux 8 Cloud Developer version ARM64

It is used in production on AWS and Azure and has been extensively tested also on GCP and Oracle Cloud.

As part of the new RonDB release we have also updated the documentation of RonDB at docs.rondb.com. Among other things it contains a new section on Contributing to RonDB that shows how you can build, test and develop extensions to RonDB. In the documentation you will also find an extensive list of the improvements made in the two new RonDB releases.

ARM64 support is still in beta phase, our plan is to make it available for production use in Q2 2022. There are no known bugs, but we want to give it a bit more time before we assign it to production workloads. This includes adding more test machines and also performing benchmarks on ARM64 VMs.

Our experience with ARM64 machines so far says that it is fairly stable, but it isn't yet at the same level as x86, it is possible to find bugs in the compilers, the support around it is however maturing very quickly and not surprising the support on Mac OS X is here leading the way since Mac OS X has fully committed its future on ARM. We have also great help of participating in the OCI ARM Accelerator program providing access to ARM VMs in the Oracle Cloud making it possible to test on Oracle Linux using ARM with both small and large VMs.

RonDB 22.01.0 comes with a set of new features:

  1. Now possible to scale reads using locks onto more threads
  2. Improved placement of primary replicas to enable
  3. All major memory areas now managed by global memory manager
  4. Even more flexibility in thread configurations
  5. Removing a scalability hog in index statistics handling
  6. Merged with MySQL Cluster 8.0.28

You can either download RonDB tarballs from https://github.com/logicalclocks/rondb or from https://repo.hops.works/master, for exact links to the various versions of the binary tarballs see Release Notes on each version.