Running Linux in Windows 11

 Most people that know me, also knows how I really don't like working with Windows. I actually worked with Microsoft products already in the 1980s, so I haven't been able to stay away from them completely.

Developing MySQL on a Windows platform haven't been so easy since building MySQL on Windows is difficult and also there are a number of issues in running even the simple test suites of MySQL.

At the same time the world is full of developers with only Windows as their development platform. So when I read about the possibility to run Linux inside Windows 11 I thought it was time to test if development of RonDB could now be done on Windows.

My 9-year old laptop needed replacement, so I went ahead and found a nice Windows laptop at a fair price. I wanted to test the very latest developments that I could find in Windows 11. Since the laptop was delivered with Windows 10 I did an unusual thing, I wanted to upgrade the OS :)

This wasn't exactly easy, took a few hours, but eventually after upgrading Windows 10 a few times and searching for a Windows 11 download I eventually found such a one using Google on the Microsoft website. After a couple of upgrades I was at the latest Windows 11 release.

Installing Linux was quite easy, it was one simple command. I installed an Ubuntu variant.

Most of the installation went ok. The graphics installation didn't work, but the installation of terminal software was good enough to test at least the testing part. For development I use graphical Vim, so this needs to wait for a working version of the graphical parts of Linux (or using a Windows editor, but likely not since they tend to add extra line feeds on the line).

Downloading the RonDB git tree went fine. Compiling RonDB required installing a number of extra packages, but that is normal and will happen also in standard Linux (build essentials, cmake, openssl dev, ncurses, bison, ..).

Running the MTR test suite also went almost fine. I had to install zip and unzip as well for this to work.

Running the MTR test suite takes about an hour and here I found a few challenges. First I had to find the parallelism it could survive. I was hoping on parallelism of 12 (which in reality for RonDB means 6 parallel tests running). But in reality it was only stable with a parallelism of 6.

However since I wasn't sitting by the Windows laptop while the test was running the screen saver decided to interfere (although I had configured it to go into screen save mode after 4 hours). Unfortunately the screen saver decided that the Linux VM should be put to sleep and this meant that all test cases running when the screen saver hit in failed. Seems like this is a new issue in WSL 2 not existing in WSL 1.

However I think that I am still happy with what I saw. Running an operating system inside another and making it feel like Linux is a part of Windows isn't an easy task. So here I must give some kudos to the development team. So if they continue working on this integration I think that I am going to get good use of my new laptop.

I must admit that I don't fully understand how they have solved the issue of running Linux inside Windows. But it definitely looks like the Linux kernel makes use of Windows services to implement the Linux services. Running top in an idle system is interesting, there is only a few init processes and a bash process. So obviously all the Linux kernel processes are missing and presumably implemented inside Windows in some fashion.

The best part is that the Linux VM configuration isn't static. The Linux VM could make use of all 16 CPUs in the laptop, but it could also allow Windows to grab most of them. So obviously the scheduler can handle both Linux and Windows programs.

Memory-wise the Linux VM defaults to being able to grow to a maximum of 80% of the memory in the laptop. However in my case top in Linux constantly stated that it saw 13.5 GB of memory in a machine with 32 GB of memory. I saw some post on internet stating that Linux can return memory to Windows if it is no longer needed. Not sure I saw this, but it is a new feature, so feel confident it will be there eventually.

So at least working with RonDB on Windows 11 is going to be possible. How exactly this will pan out I will write about in future worklogs. At least it is now possible that I can do some development in Windows, it was more than 30 years ago I last had a development machine with a Microsoft OS, so to me, Linux on Windows is definitely making Windows as a platform a lot more interesting.

My development environments have shifted a bit over the years. It started with a mix of Microsoft OSs and Unix and some proprietary OSs in the 80s. In the 90s I was mostly working in Solaris on Sun workstations. Early 2000's I was working with Linux as development machine. But since 2003 I have been working mostly on Mac OS X (and of course lots of test machines on all sorts of platforms).

My endeavour to build MySQL on Windows

I recently worked a lot on getting my local Windows box to compile Windows.
I got a lot of help from the below link that took me a few steps towards the
solution.
http://www.chriscalender.com/resolving-the-bison-exe-m4-invalid-argument-error-when-building-mysqlmariadbxtradb-on-windows/

I had to save away my bison installation, uninstall it and then also
go into a registry editor and remove the various links to the Start Menu referring to
some help for Bison that had spaces in the registry. Resolving this meant that I could
manually run bison on the files in the mysql tarball and get the desired result. But it
was still not enough, I still had the same error when running a full automatic compile.

So then I found the following link coming to my rescue.
http://venublog.com/2010/02/07/changing-mysql-parser-code-on-windows-build-breaks-due-to-bison/

With the aid of this link I realised that I had to make sure that the correct bison
executable was started. So in order to get this to happen I ensured that C:\GnuWin32
was the very first part in PATH.

With this final touch I was finally able to get my Windows builds of MySQL and
MySQL Cluster to work like a charm. I still don't fully understand why the
second step was necessary, but it worked which is good enough for me :)

Do I understand why Windows supports spaces in file names, no.

New threadpool design

In MySQL 6.0 a threadpool design was implemented based on
libevents and mutexes.

This design unfortunately had a number of deficiences:
1) The performance under high load was constrained due to a global
mutex protecting libevent (see BUG#42288).

2) The design had no flexibility to handle cases where threads were
blocked due to either locking or latches. E.g. a thread held up by a
table lock will be kept in the threadpool as an active thread until
the table lock is released. If all threads are blocked in this state,
it's easy to see that also any query that want to release the table
lock cannot be processed since all threads in the thread pool are
blocked waiting for the table lock (see BUG#34797).

3) The design is intended to support very many connections but
didn't use the most efficient methods to do this on Windows.
libevent uses poll on Windows which isn't a scalable API when
there are thousands of connections.

Also in all of the benchmarking with MySQL it's been clear that
performance of MySQL often drops significantly when there are too
many threads hitting the MySQL Server. We have seen vast
improvements of this the last year and there are some additional
improvements of this in the pipeline for inclusion into the next
MySQL milestone release. However the basic problem is still there,
that too many waiters in the queue can lead to various performance
drop off, one reason for such drop offs can be when mutex waits
starts to timeout in InnoDB.

So actually when we're looking at the threadpool design now, we're
aiming at solving two issues in one. The first is to remove this
scalability dropoff at high thread counts and the second is to
efficiently handle MySQL servers with thousands of connections.
Threadpool also enables us to have more control over on which
CPU threads are scheduled to execute on. We can even dynamically
adapt the CPU usage to optimize for lower power consumption by
the MySQL Server with a clever threadpool design.

We're currently in the phase of experimenting with different
models, however we opted for a design based around usage of epoll
on Linux, eventports on Solaris and kqueue for FreeBSD and
Mac OS X. We will also make a poll-based variant work mostly for
portability reasons although it's scalability won't be so great.
For Windows we're experimenting with some Windows specific
API's such as the IO Completion API.

The code to support thread pooling in MySQL is actually very
small so it's easy to adapt the code for a new experiment.

Last week we found a model that seems to work very fine.
The benchmarks shows that the performance on 1-32 threads is
around 97-103% of one thread per connection performance. When
we go beyond 32 threads the thread pool gains more and more,
it's getting to about 130% at 256 threads and reaches 250%
better performance on 1024 threads. However this model still
have the problem of deadlocks, so there is still some work on
refining this model. The current approach we have is fixing
the deadlock problem but removes about 10-15% of the
performance on lower number of threads. We have however
numerous ideas on how to improve this.

The basic idea with our current approach is to use thread groups,
where each group works indepently of other groups in handling a
set of connections. We're experimenting with the number of
threads per group and also how to handle the situation when the
last thread in the group is getting ready to execute a query.

Compared to maximum performance around 32 threads we reach
about 67% of this performance also on 1024 concurrently active
threads. The drop off 33% is expected since there is some
additional load when we reach an overload situation to ensure
that the proper thread is handling the task. At low number of
threads it's possible to immediately schedule the current worker
thread to work on the query, but in the overload situation there
is some queueing and context switching needed to handle the
situation. However the overhead at overload is constant, so it
doesn't worsen when the number of threads goes to a very high
number.

To handle the problems with blocked threads, we will implement a
new part of the storage engine API and API towards the MySQL
Server where the MySQL Server and the storage engines can
announce that they're planning to go inactive for some reason.
The threadpool will however handle the situation even if a thread
goes to sleep without announcing it, it will simply be more
performant if the announcement comes in those situations.

The new MySQL development model with milestone release is a
vital new injection to the MySQL development leading to the
possibility of making new features available to the MySQL
community users in an efficient manner without endangering the
quality of the MySQL Server. There is a very strict quality model
before approving any new feature into a milestone release.
The 6.0 thread pool design would not meet this strict quality
model. The new design must meet this strict quality model before
being accepted although we have good hopes for this to happen.