Releasing iClaustron API, a new Rust NDB API with a C and a Rust API towards RonDB 26.10.0

We have released version 1.0.0 of the iClaustron API. The github tree is at https://github.com/logicalclocks/iclaustron
Training material for Rust is available here
Training material for C is available here

Extensive training material on RonDB is available here

Some takeaways:

1. MIT licensed
2. Receiver thread only routing messages => Scales to 3x more CPUs
3. Capable of doing 9M key lookups per second on a Mac M5 Pro laptop running cluster and API on the laptop
4. Cost of key lookup can be as low as 500ns per row in iClaustron API
5. 0 memory allocation for a normal key lookup
6. Currently supports Key lookups and Scan operations

PeakAIO contributes support of 8k API nodes to RonDB 26.02.11

PeakAIO develops pNFS Lattice, a new parallel file system using RonDB as a metadata server. To support larger clusters they needed more API nodes. RonDB 25.10 added support for 2039 API nodes, now 26.02.11 when released will support 8191 API nodes.

Since originally RonDB supported 255 API nodes, changing to 8191 API nodes caused a number of memory segments to grow significantly. So part of the work was related to various improvements on memory usage, performance in management server. So the consequence in using RonDB for smaller clusters is that they will use less memory than before.

If you want to understand more about pNFS Lattice and their use of RonDB there are two presentations at SNIA Developers conference on 28 September in Santa Clara First presentation Second presentation.

Two-level hashing for improved scans in distributed databases

In a distributed database using hash partitioning for the tables there is usually two options when you scan a table. Either you know the partition key and only scan a single partition, or you scan all partitions.

Assume you are scanning for hundreds of rows. It could be a table containing the files in a directory as an example, or it could be the events from a user. In both cases both variants of the scan have issues.

The single partition to scan is very efficient, but it has no parallelism and thus can have long latency when looking for hundreds of rows.

The all partition to scan is very inefficient when the number of partitions is large, but it will typically have low latency if the load on the cluster is low (stragglers are not a big problem in RonDB, due to its real-time scheduler)."

It would be nice to find a combination that delivers low latency at high efficiency. This is a problem that has been on my mind for decades, the solution I think is fairly elegant.

In RonDB we are adding a new feature called two-level hashing. What this means is that the hashing is done in two steps. Assume your primary key is two columns, one is the event id and the second is a timestamp. In this case you first hash on the event id, this gives you a part of the hash key. Second you hash on the timestamp. Last you put the two hashes together into a single hash key. Another example is a primary key consisting of a directory id and a file name that would be useful in a metadata server of a file system.

For key lookups it is a new way of calculating the partition to store the row, but each row can only end up in a single partition.

However, in the case of a scan that want to get all rows for a certain event id you need to scan a subset of the partitions. Hashing the event id gives you the first potential partition rows can be stored in, to find all rows of the event id you have to scan a number of partitions starting from the first. We call this the fanout. So if the fanout is 8, it means you have to scan 8 partitions to find all rows for that event id you are looking for.

We ran a benchmark where we compared a two-level hashing with a traditional hashing on the primary key. The number of partitions was 96. The benchmark executed index scans on the event id and used a limit of 100 or 200.

We found that by using a fanout of 8 and a limit of 100 the throughput doubled and the latency decreased to half compared to the traditional approach. We tested with various fanouts and 8 was the optimal number. However increasing it to 12 only decreased throughput by less than 3%. So a fanout of 16 is better at handling larger limits. The exact fanout that should be chosen is of course dependent on the application. But the experiment showed that the feature had a significant impact on the throughput and latency of cases where your application does a lot of scanning for hundreds of rows.

We can see that this can benefit our applications both in metadata servers for file systems and for feature stores storing event rows where scanning for a few hundred rows is common. Using a Partition Key that is part of the primary key turns out to be a special case of this where the fanout is 1.

Experiences from the new wave of AI programming

How we built RonSQL in RonDB with Claude and Codex

A false start

A few years ago at Hopsworks we made an early attempt at using ChatGPT for coding. It turned out to be an expensive mistake: the generated code cost us several months of rewriting. So when I came back to AI programming this year, I did so with a fair amount of skepticism.

Trying again — the small wins

In January we tried again, and the first success came from an unexpected place. Our marketing manager put together a simple RonDB client in about two hours. That caught my interest. I took the code over and, in a few more hours, had extended it significantly.

Encouraged, I tried something closer to real engineering: extending our REST API server so that it could handle not only batched key reads, but also batched key inserts, writes, and deletes. The original code had taken significant effort to write; the extension took just two days. Clearly, extending existing code was something Claude and Codex could do well.

The big bet: a ten-year TODO item

With those successful experiments behind me, I realised this new tooling might be extremely useful for developing genuinely new features in RonDB.

Something our customers have wanted for a long time is more complex queries in real-time AI inferencing. Supporting that requires CTEs and parallel join aggregation — a capability that has sat on the RonDB TODO list for more than ten years. It is very complex development work; with conventional coding I would estimate it as a two-year project, at least.

What this really amounts to is adding subsets of complex SQL support to a key-value store. RonDB sits in the same category as key-value stores such as Redis and DynamoDB, which are traditionally used for exactly this kind of real-time, low-latency serving — but which have no support for complex SQL queries at all. Bringing CTEs and parallel join aggregation into that world is what makes the effort both unusual and worthwhile.

Would it be feasible now, with AI programming? I decided to find out. This post shares what I learned over the past five months.

The short version: today we released RonDB 26.04.1, which ships beta support for RonSQL with CTEs and pushdown join aggregation.

The development model

From the start it was clear that the proven development model still applied. You begin with a high-level plan, move to a detailed implementation plan, and only then implement it step by step.

We went through this loop several times. Each plan had at least 10–20 phases, and sometimes many more.

Claude vs. Codex

Working with both models, my conclusion is that their strengths are complementary. Claude is better at keeping track of plans and remaining work. Codex is often better at solving hard problems, but it is sloppier about maintaining the plan and can lose track of what to do next. So for the most part I let Claude own plan maintenance and the less complex tasks, and handed the hard bugs to Codex.

Massive testing across four layers

One thing that struck me immediately is that AI programming makes it possible to generate massive amounts of unit tests — even for the distributed cases that are normally painful to test.

RonSQL query execution spans four layers in RonDB:

• LDM (Local Data Manager) — the lowest layer, six modules that handle local query execution, local recovery, and local checkpointing.
• TC (Transaction Coordinator) — coordinates join execution for complex queries. Partial results flow from the first tables in the join to the later ones. The TC drives execution from a Query Tree: essentially a program that specifies how to run the query, including interpreted code to be sent down to the LDMs.
• NDB API — defines the Query Tree through an API. Applications can use it directly, though that is uncommon because it is low-level.
• RonSQL — the top layer, which translates SQL into NDB API calls. The NDB API builds a Query Tree and sends it to the TC; the TC distributes execution across the RonDB nodes, and each part runs on the LDM in every node.

Normally you write test programs in RonSQL, because writing and maintaining programs against the LDM, TC, and NDB API layers by hand is very hard. AI programming changes that. It became feasible to develop and test the LDM layer first, before touching the TC layer, and the TC layer before touching the NDB API layer. This made it much easier to reach a working implementation across all four layers quickly.

The asynchronous challenge

AI programming works very well on the kind of sequential code the models have seen in training. But RonDB uses an asynchronous programming model, and here the models sometimes needed firm guidance to understand how things actually work. Things that were obvious to me were sometimes completely invisible to the model — but it was usually easy enough to teach it through the prompts. The reverse was true just as often: the models generate new code far faster than a human can think.

How do you verify the AI got it right?

So how do you verify that the generated code is correct? Here is the model I try to follow.

Phase 1 — Build to the plan

Follow the high-level plan, then the implementation plan, then implement step by step. In this phase you are the architect: you tell the model what to do at a high level, and for the most part you act as an operator who accepts its suggestions. I found it essential not to let the model change any code without stating exactly what it changed and how. With Claude this was the default behaviour; Codex had to be kept on a tight leash to stop it running ahead and generating code without checking whether it was correct. Seeing every diff gives you at least a working idea of what the new code does — and this is the phase to generate lots of tests.

Crucially, each step includes writing at least one test case and verifying that the step has been successfully executed before moving on to the next one. A step is not “done” until there is a passing test that proves it. This keeps the implementation grounded and stops small mistakes from compounding across the many phases of a plan.

Phase 2 — Review

Once the steps are done, review all the new code. I focus mostly on the code that affects actual execution, and only glance at test cases and plans. I used the model to generate documentation — with images and signalling diagrams — to better understand the new code. This phase is where you ensure the model follows RonDB's memory model and correctly handles node failures and other failures in query execution. The models also tend to generate a lot of duplicated code, so removing duplicated code paths is an essential step — some of these removed thousands of lines of production code that would otherwise have had to be maintained.

Phase 3 — Stress the feature with tests

We then wrote CTE test cases without regard to whether they were supported. When a feature was not supported, we either fixed it or tagged it as unsupported.

What remains is to minimise query latency and to keep reviewing the generated code.

Working in parallel

One final observation: while the model is thinking, there is plenty of time to work on something else. Most of the time I had two to four projects running in parallel. That is how, alongside RonSQL, prototypes appeared for a JIT compiler for the RonDB interpreters, for using fibers in RonDB, and for Deadlock Discovery. Only Deadlock Discovery is in the source tree so far, and it is disabled by default. All three are still work in progress.

Takeaway

Looking back, a feature that had been on the TODO list for over a decade — and that I would have scoped as a two-year effort — reached beta in five months. The work did not disappear; it changed shape. My role shifted from writing the code to architecting it, guiding the models, reviewing the result, and above all testing it. That, more than raw code generation, is where the new wave of AI programming earned its place in how we build RonDB.