RonDB has now grown up to the same level of memory management as you find in expensive commercial DBMSs like Oracle, IBM DB2 and Microsoft SQL Server.
Today I made the last development steps in this large project. This project started with a prototype effort by Jonas Oreland already in 2013 after being discussed for a long time before that. After he left for Google the project was taken over by Mauritz Sundell that implemented the first steps for operational records in the transaction manager.
Last year I added the rest of the operational records in NDB. Today I completed the programming of the final step in RonDB. This last step meant moving around 30 more internal data structures towards using the global memory manager. These memory structures are used to represent meta data about tables, fragments, fragment replicas, triggers and global replication objects.
One interesting part that is contained in this work is a malloc-like implementation that interacts with all record-level data structures that is already in RonDB to handle linked list, hash tables and so forth for internal data structures.
So after more than 5 years it feels like a major step forward in the development of RonDB.
What does this mean for a user of RonDB? It means that the user won't have to bother much with memory management configuration. If RonDB is started in a cloud VM, it will simply use all memory in the VM and ensure that the memory is handled as a global resource that can be used by all parts of RonDB. This feature is exactly existing already in RonDB 21.04. What this new step means is that the memory management is even more flexible, there is no need to allocate more memory than needed for meta data objects (and vice versa if more memory is needed, it is likely to be accessible).
Thus memory can be used for other purposes as well. Thus the end result is that more memory is made available in all parts of RonDB, both to store data in it and to perform more parallel transactions and more query handling.
Another important step is that this step opens up for many new developments to handle larger objects in various parts of RonDB.
In later blogs we will describe how the memory management in RonDB works. This new development will either appear in RonDB 21.10 or in RonDB 22.04.
2 comments:
Better late than never?
Hi Jonas,
Yep, Mauritz started out with your patch serie, but he reversed the order, so the final
step will be to handle the meta data stuff. This will arrive soon in RonDB 21.10.2 which
should be the final step for our journey towards automatic memory management.
Partial LCPs is probably resembling your ideas on Local LCP, we still maintain the
LCP checkpoint protocol but mostly ignore the signals sent between the nodes.
All the information about how to restore the checkpoints is available locally.
Quite a challenge to drop the entire LCP protocol with regard to upgrade.
Post a Comment