Experiences from the new wave of AI programming

How we built RonSQL in RonDB with Claude and Codex

A false start

A few years ago at Hopsworks we made an early attempt at using ChatGPT for coding. It turned out to be an expensive mistake: the generated code cost us several months of rewriting. So when I came back to AI programming this year, I did so with a fair amount of skepticism.

Trying again — the small wins

In January we tried again, and the first success came from an unexpected place. Our marketing manager put together a simple RonDB client in about two hours. That caught my interest. I took the code over and, in a few more hours, had extended it significantly.

Encouraged, I tried something closer to real engineering: extending our REST API server so that it could handle not only batched key reads, but also batched key inserts, writes, and deletes. The original code had taken significant effort to write; the extension took just two days. Clearly, extending existing code was something Claude and Codex could do well.

The big bet: a ten-year TODO item

With those successful experiments behind me, I realised this new tooling might be extremely useful for developing genuinely new features in RonDB.

Something our customers have wanted for a long time is more complex queries in real-time AI inferencing. Supporting that requires CTEs and parallel join aggregation — a capability that has sat on the RonDB TODO list for more than ten years. It is very complex development work; with conventional coding I would estimate it as a two-year project, at least.

What this really amounts to is adding subsets of complex SQL support to a key-value store. RonDB sits in the same category as key-value stores such as Redis and DynamoDB, which are traditionally used for exactly this kind of real-time, low-latency serving — but which have no support for complex SQL queries at all. Bringing CTEs and parallel join aggregation into that world is what makes the effort both unusual and worthwhile.

Would it be feasible now, with AI programming? I decided to find out. This post shares what I learned over the past five months.

The short version: today we released RonDB 26.04.1, which ships beta support for RonSQL with CTEs and pushdown join aggregation.

The development model

From the start it was clear that the proven development model still applied. You begin with a high-level plan, move to a detailed implementation plan, and only then implement it step by step.

We went through this loop several times. Each plan had at least 10–20 phases, and sometimes many more.

Claude vs. Codex

Working with both models, my conclusion is that their strengths are complementary. Claude is better at keeping track of plans and remaining work. Codex is often better at solving hard problems, but it is sloppier about maintaining the plan and can lose track of what to do next. So for the most part I let Claude own plan maintenance and the less complex tasks, and handed the hard bugs to Codex.

Massive testing across four layers

One thing that struck me immediately is that AI programming makes it possible to generate massive amounts of unit tests — even for the distributed cases that are normally painful to test.

RonSQL query execution spans four layers in RonDB:

• LDM (Local Data Manager) — the lowest layer, six modules that handle local query execution, local recovery, and local checkpointing.
• TC (Transaction Coordinator) — coordinates join execution for complex queries. Partial results flow from the first tables in the join to the later ones. The TC drives execution from a Query Tree: essentially a program that specifies how to run the query, including interpreted code to be sent down to the LDMs.
• NDB API — defines the Query Tree through an API. Applications can use it directly, though that is uncommon because it is low-level.
• RonSQL — the top layer, which translates SQL into NDB API calls. The NDB API builds a Query Tree and sends it to the TC; the TC distributes execution across the RonDB nodes, and each part runs on the LDM in every node.

Normally you write test programs in RonSQL, because writing and maintaining programs against the LDM, TC, and NDB API layers by hand is very hard. AI programming changes that. It became feasible to develop and test the LDM layer first, before touching the TC layer, and the TC layer before touching the NDB API layer. This made it much easier to reach a working implementation across all four layers quickly.

The asynchronous challenge

AI programming works very well on the kind of sequential code the models have seen in training. But RonDB uses an asynchronous programming model, and here the models sometimes needed firm guidance to understand how things actually work. Things that were obvious to me were sometimes completely invisible to the model — but it was usually easy enough to teach it through the prompts. The reverse was true just as often: the models generate new code far faster than a human can think.

How do you verify the AI got it right?

So how do you verify that the generated code is correct? Here is the model I try to follow.

Phase 1 — Build to the plan

Follow the high-level plan, then the implementation plan, then implement step by step. In this phase you are the architect: you tell the model what to do at a high level, and for the most part you act as an operator who accepts its suggestions. I found it essential not to let the model change any code without stating exactly what it changed and how. With Claude this was the default behaviour; Codex had to be kept on a tight leash to stop it running ahead and generating code without checking whether it was correct. Seeing every diff gives you at least a working idea of what the new code does — and this is the phase to generate lots of tests.

Crucially, each step includes writing at least one test case and verifying that the step has been successfully executed before moving on to the next one. A step is not “done” until there is a passing test that proves it. This keeps the implementation grounded and stops small mistakes from compounding across the many phases of a plan.

Phase 2 — Review

Once the steps are done, review all the new code. I focus mostly on the code that affects actual execution, and only glance at test cases and plans. I used the model to generate documentation — with images and signalling diagrams — to better understand the new code. This phase is where you ensure the model follows RonDB's memory model and correctly handles node failures and other failures in query execution. The models also tend to generate a lot of duplicated code, so removing duplicated code paths is an essential step — some of these removed thousands of lines of production code that would otherwise have had to be maintained.

Phase 3 — Stress the feature with tests

We then wrote CTE test cases without regard to whether they were supported. When a feature was not supported, we either fixed it or tagged it as unsupported.

What remains is to minimise query latency and to keep reviewing the generated code.

Working in parallel

One final observation: while the model is thinking, there is plenty of time to work on something else. Most of the time I had two to four projects running in parallel. That is how, alongside RonSQL, prototypes appeared for a JIT compiler for the RonDB interpreters, for using fibers in RonDB, and for Deadlock Discovery. Only Deadlock Discovery is in the source tree so far, and it is disabled by default. All three are still work in progress.

Takeaway

Looking back, a feature that had been on the TODO list for over a decade — and that I would have scoped as a two-year effort — reached beta in five months. The work did not disappear; it changed shape. My role shifted from writing the code to architecting it, guiding the models, reviewing the result, and above all testing it. That, more than raw code generation, is where the new wave of AI programming earned its place in how we build RonDB.

RonSQL: a new SQL engine for RonDB with predictable low latency and CTEs

Today we released RonDB 26.04.1, a beta release. It contains a lot of new features, but the most interesting one is that RonSQL now supports pushdown join aggregation and CTEs, so that complex queries run with low, predictable latency.

RonDB has always been able to answer complex queries through a MySQL Server. The problem with that path is predictability. The application asks for an answer, but it has no guarantee about how fast that answer arrives: the MySQL optimizer picks a plan that may or may not parallelise the query across the RonDB data nodes, and a plan that looks fine on a small table can fall off a cliff as the data grows.

RonSQL takes a different contract. The rule is simple:

Anything RonSQL accepts can be pushed down to the RonDB data nodes for parallel execution.

If a query parses and plans in RonSQL, it runs as a parallel pushdown — there is no fallback to a slow, single-threaded plan. That means the latency of a complex query is something the application can actually reason about up front, instead of discovering it in production.

Why this matters: Feature Stores

RonSQL grew out of the needs of AI applications built on Feature Stores, and in particular on-demand (real-time) transformations in Hopsworks.

Traditionally an online Feature Store only does primary-key lookups. To keep those lookups fast, every feature has to be pre-computed and written back before serving. That works, but it has two costs:

1. Stale features. A feature is only as fresh as the last batch job that recomputed it. The events from the last few seconds — often the most predictive ones for fraud, recommendations, or anomaly detection — are not yet reflected.
2. Expensive BLOB packing. A common trick is to pack a range of values into a single BLOB (often Avro-encoded) so a whole feature group can be fetched in one lookup. But every change to any value means re-encoding the whole BLOB, which may hold hundreds of values.

RonSQL attacks both problems:

• Fresh features on the fly. Instead of pre-computing aggregations, you stream raw rows into RonDB and let RonSQL aggregate them at query time. A row inserted a second ago is included immediately, so the feature reflects what just happened.
• Index scans instead of BLOBs. Instead of packing values into a BLOB and re-encoding on every change, you store the values as ordinary rows and let RonSQL read them with an index scan. Updates become simple inserts and deletes — and deletes are usually handled for you by RonDB's row-level TTL, so old data ages out without any application code.

CTEs (Common Table Expressions, the SQL WITH clause) are what let you combine these two ideas in a single, readable query: aggregate the fresh fact rows in a CTE, then join the result against your normalised dimension tables.

A worked example: real-time card-fraud features

Consider a fraud-scoring model. At inference time it needs a feature vector for one card, computed over that card's most recent activity. The raw transactions arrive continuously and are inserted straight into RonDB:

-- Fact table: one row per card transaction, inserted in real time.
CREATE TABLE txn (
  txn_id       BIGINT       NOT NULL,
  cc_num       BIGINT       NOT NULL,   -- card / account identifier
  merchantkey  INT          NOT NULL,   -- references merchant.m_merchantkey
  amount       INT          NOT NULL,   -- minor units (cents)
  txn_time     DATETIME(6)  NOT NULL,
  is_declined  TINYINT      NOT NULL,
  PRIMARY KEY USING HASH (txn_id),
  -- Ordered index: range-scan one card's recent activity cheaply.
  INDEX idx_card_time (cc_num, txn_time)
) ENGINE=NDB
  COMMENT='NDB_TABLE=TTL=604800@txn_time';  -- auto-expire rows after 7 days

-- Small dimension table: replaces a per-card Avro BLOB of merchant attributes.
CREATE TABLE merchant (
  m_merchantkey INT          NOT NULL,
  m_category    VARCHAR(16)  NOT NULL,
  m_risk_score  INT          NOT NULL,
  PRIMARY KEY USING HASH (m_merchantkey)
) ENGINE=NDB;

Step 1 — a fresh feature vector with a single scan

The simplest on-demand feature is a scalar aggregate over the card's last hour of transactions. No pre-computation, no BLOB — just an index range scan that includes whatever was inserted milliseconds ago:

SELECT
  COUNT(*)                                          AS txns_1h,
  SUM(amount)                                       AS amount_1h,
  MAX(amount)                                       AS max_amount_1h,
  AVG(amount)                                       AS avg_amount_1h,
  SUM(CASE WHEN is_declined = 1 THEN 1 ELSE 0 END)  AS declines_1h
FROM txn
WHERE cc_num = 4716253018273645
  AND txn_time >= DATE_SUB('2026-06-29 14:30:00', INTERVAL 1 HOUR);

RonSQL turns the WHERE into an ordered-index range scan on idx_card_time — it touches only this card's last hour — and pushes the COUNT/SUM/MAX/AVG and the CASE expression down to the data nodes, which aggregate in parallel and return a single row.

Step 2 — combining fresh aggregation with a dimension join, using a CTE

Now suppose the model wants spend broken down by merchant category. The category does not live on the transaction — it lives on the merchant dimension. The classic Feature Store approach would denormalise the category into a packed BLOB per card. With RonSQL we keep the data normalised and join at query time:

WITH spend_by_merchant AS (
  SELECT merchantkey AS m,
         SUM(amount) AS spend,
         COUNT(*)    AS txns
  FROM txn
  WHERE cc_num = 4716253018273645
    AND txn_time >= DATE_SUB('2026-06-29 14:30:00', INTERVAL 1 HOUR)
  GROUP BY merchantkey
)
SELECT m.m_category                  AS category,
       SUM(spend_by_merchant.spend)  AS spend_last_hour,
       SUM(spend_by_merchant.txns)   AS txns_last_hour
FROM merchant AS m
JOIN spend_by_merchant ON spend_by_merchant.m = m.m_merchantkey
GROUP BY m.m_category;

This query is easy to reason about, top to bottom:

1. The CTE spend_by_merchant runs an ordered-index range scan on idx_card_time, restricted to one card over the last hour — the only large table in play. The data nodes aggregate SUM(amount) and COUNT(*) grouped by merchantkey, returning just a handful of rows (one per merchant the card touched in that hour).
2. The join attaches the merchant attributes. m_merchantkey is the primary key of merchant, so each row is resolved with a cheap primary-key lookup rather than another scan. merchant is a small dimension table.
3. The outer query re-aggregates the joined result by m_category, producing one row per merchant category — a compact, model-ready feature vector.

Every stage is a pushdown, and stages such as the index scan and the lookups run in parallel across the data nodes. We could even execute several CTEs in parallel. Because RonSQL guarantees the whole thing pushes down, the latency is bounded and predictable — which is exactly the contract a real-time inference path needs.

Running a RonSQL query

RonSQL is reachable two ways:

• REST (RDRS). The RonDB REST server exposes a RonSQL endpoint, which is the path used by online serving. It even keeps a built-in latency histogram so you can watch the predictable-latency promise hold in production. The rondb-cli shell sends a line straight to it with the RONSQL prefix.
• ronsql_cli. A standalone client for scripting and experimentation. It reads a query from --execute, --execute-file, or stdin and can emit results as JSON (ideal for a feature vector) or TEXT.

Both paths support EXPLAIN. Prefixing a query with EXPLAIN shows the chosen pushdown plan — which index drives each scan, which joins become lookups, and where the aggregation happens — so “will this be fast?” is a question you answer before you ship, not after.

What RonSQL supports today

RonSQL is a read-only, aggregation-focused SQL subset designed so that everything it accepts can be pushed down:

• Statements: SELECT only (plus EXPLAIN). No DDL/DML.
• CTEs: multiple, comma-separated WITH clauses (non-recursive); a CTE can be joined as a child or used as the driving table.
• Joins: INNER JOIN, LEFT [OUTER] JOIN, self-joins, and comma cross-joins over scalar CTEs. Equi-join conditions, including composite keys (a.x = b.x AND a.y = b.y).
• Filtering: rich WHERE — = <> < <= > >=, LIKE, IN (list), IS [NOT] NULL, AND/OR/XOR/NOT, arithmetic, bitwise ops, and CASE WHEN.
• Subqueries: EXISTS, IN (subquery), and scalar subqueries.
• Aggregates: COUNT(*), COUNT(expr), SUM, MIN, MAX, AVG.
• Grouping & shaping: GROUP BY (multi-column, any table), HAVING, ORDER BY ASC/DESC, LIMIT.
• Expressions: arithmetic, CASE WHEN, GREATEST/LEAST, and date/time functions DATE_ADD, DATE_SUB, EXTRACT, INTERVAL.
• Index hints: FORCE INDEX, USE INDEX, IGNORE INDEX.

Why express features in SQL at all?

Because the Feature Store has to compute the same feature in two very different settings. Batch training and batch inference run on engines like Spark SQL and DuckDB — both batch query engines, chosen for different characteristics (Spark scales the work across a cluster for very large datasets; DuckDB runs embedded and is hard to beat on a single node for moderate data). Online serving runs on RonSQL, computing the feature fresh at inference time. When all of them speak SQL, the same feature logic can be expressed as the same query text on each engine, which eliminates a notorious source of training/serving skew — features that subtly differ between the model's training data and what it sees live at inference.

Where RonSQL goes next

RonSQL is already useful, but there is a clear roadmap, much of it driven directly by Feature Store needs:

• Distinct-count features. COUNT(DISTINCT ...), and an approximate variant (HyperLogLog), to answer “how many distinct merchants / devices / countries in the last hour?” — a staple fraud signal. DISTINCT and OFFSET more generally.
• More aggregate functions. STDDEV and VARIANCE (for z-score features), and GROUP_CONCAT.
• Point-in-time correctness (“time travel”). As-of joins so the same RonSQL query can reconstruct a feature's value at a historical timestamp for training, exactly matching what online serving would have returned — closing the skew gap completely.
• Richer query shapes. Derived tables / subqueries in FROM, UNION, RIGHT/FULL OUTER JOIN, and recursive CTEs for hierarchy/graph features.
• Vector / embedding pushdown. Top-K nearest-neighbour search pushed to the data nodes, as embeddings increasingly live alongside scalar features.
• Cost-based join ordering. The planner currently joins left-to-right; reordering based on table/index statistics would make more queries fast by default.
• Continuous / materialised features. Incrementally maintaining a CTE's result as new rows arrive, blurring the line between on-demand and pre-computed features.

The core contribution stays the same: predictable low latency for complex queries over fresh data, expressed in portable SQL — exactly what an online Feature Store needs to serve fresh, skew-free features to an AI model.

RonDB gets a RDMA transporter supporting pNFS Lattice

Today, PEAK-AIO announced pNFS Lattice, a metadata server for Parallel NFS. It is running on top of RonDB using an RDMA transporter.

Now you might wonder, does RonDB have an RDMA transporter? A few weeks ago the answer would have been no. It has been on the TODO list for more than a decade, but it was not prioritized. However, a few weeks ago Eyal Lemberger from PEAK-AIO contributed an RDMA transporter to RonDB. Me and Eyal worked on the contribution for a few weeks and it was stable enough to integrate into RonDB 26.02, our latest stable release.

In the normal builds, the RDMA transporter is disabled, so it requires manual building to integrate it. The aim is to integrate the RDMA transporter into the RonDB release planned for the end of the year 2026.

3x Latency Improvement

RDMA improved the latency of metadata operations by a factor of 3. Actually, this brings RonDB back to its roots. The very first transporter that NDB Cluster (RonDB is a fork of NDB Cluster) had was a Dolphin SCI transporter. I have worked with Dolphin for more than 30 years. Today Dolphin technology can be used with a normal TCP/IP transporter using normal sockets.

The Wakeup Cost Challenge

The biggest challenge of an RDMA transporter is the same as the challenge of the shared memory transporter. It is about how to wake up the receiving process—it is important not to pay the wakeup cost for every signal.

In RDMA, one can send batches and the wakeup is handled by the network card. Thus, the sender has no cost attached to sending other than writing to the receivers memory. The wakeup cost is only at the receiving side and is integrated in the network card, so the only cost is an occasional interrupt to ensure that the receiver is awake.

Great to see RonDB coming to good use and the open-source license making the technology available to many. Many thanks to PEAK-AIO for the contribution.

Developing RonDB with Claude: Lessons from 30 Years of Database Engineering

After 30 years of developing MySQL NDB Cluster and RonDB, a new tool has fundamentally changed how I write code. Here is what I have learned about working with Claude on a large-scale distributed database project.

A New Era for Database Development

A while ago I got a new tool that completely changes the way I develop code. I have worked on developing MySQL NDB Cluster — and now the fork RonDB — for more than 30 years. Over that time, many of the original findings about software engineering have shifted. In the 1990s, I learned that unit testing was important. However, I quickly discovered that in a startup with limited resources, it was not feasible to write unit tests for distributed database functionality. I even wrote a tool for generating distributed unit test programs, but the overhead remained too high.

With Claude, this equation completely changes. I can now modify 1,000 lines of code in a day — often more — and in parallel, instruct Claude to write 3–5x as many lines of test code for comprehensive unit testing. Claude does not just improve my coding productivity; it also makes the path to high-quality code faster by enabling test coverage that was previously impractical.

Will AI Cause Unemployment?

Some ask a philosophical question: does this mean the world will see unemployment due to AI coding tools? Personally, I think not. For me, it simply means that features I have been wanting to build for 20+ years are now suddenly possible. The only reason for unemployment would be if humanity ran out of ideas for what to develop next. I do not believe that will ever happen — just look into space and realise that God's creation is far too vast for us to explore, even with 1,000x more compute power than we have today. There will always be a new thing to understand and develop around the next corner.

A Word of Caution

Not all my experiences with AI coding have been positive. We had a REST API server written in Go that needed to be ported to C++. The AI performed a straightforward translation, but this created significant performance issues and produced code that was unreadable unless you had studied every new C++ feature in the latest standard. The translation took two months; fixing the resulting issues took six months. In retrospect, writing the C++ implementation from scratch would likely have been more efficient than using AI translation.

The lesson: AI works best when you guide it with clear architectural direction, not when you use it for mechanical translation without oversight.

Getting Started: The RonDB CLI

My first real attempt at using Claude for RonDB was creating a CLI tool. A colleague initally created it and I developed it further. I realised how well-suited this type of boilerplate code was for AI assistance. I extended our REST API, added new CLIs for Rondis, the REST API, and even a MySQL client interface. This was straightforward work — it would have been fairly easy even without Claude — but it still would have taken two to three months. With Claude, it was done within a week.

The Big Challenge: Pushdown Join Aggregation

Encouraged by the CLI experience, I decided to tackle something far more ambitious. For over ten years, I had wanted to develop Pushdown Join Aggregation in NDB/RonDB. This feature would allow complex join queries with aggregation to execute directly in the data nodes rather than pulling data up to the MySQL Server. However, it was a task that would have taken a year or more, so it never rose high enough on the priority list. With Claude, I estimated I could complete it in one to two months.

Background

NDB/RonDB already had two key building blocks in place. First, Pushdown Join has been supported for a long time, enabling complex join queries to run with higher parallelism and improving performance by more than 10x compared to executing them via the MySQL Server. Second, we developed RonSQL, which supports pushdown aggregation on a single table. Many pieces were already there, but extending from single-table aggregation to complex join queries was still a major undertaking.

The Development Approach

In his blog post How I Use Claude Code, Boris Tane describes the importance of planning your work before handing it to Claude. That is definitely true, but for a task of this complexity, even more structure was needed.

Divide and Conquer: Four Modules

The task naturally breaks down into four modules:

1. Local Database — Where the actual aggregation happens and intermediate results are stored during query execution
2. Coordinator — Distributes query fragments across nodes and coordinates their execution
3. API — The application interface through which clients interact with the system
4. SQL — Transforms SQL statements into query plans sent to the NDB API and coordinator

Figure 1: The five-layer architecture of Pushdown Join Aggregation in RonDB

Each module had to be developed separately. I started with the local database part, where the core aggregation logic lives.

Architecture First, Then Implementation

I began by asking Claude for an architecture description, providing the fundamentals of how I wanted the aggregation handling to work — something I had been thinking about for many years. Claude produced a phased development plan. The original plan contained six phases; by the end, I had gone through 15–20 phases with constant refinements.

Figure 2: The iterative development workflow when working with Claude

From Implementation to Testing

After about two to three days, the local database implementation was ready. At that point I realised that Claude made it possible to unit test the new code — something that would have been prohibitively expensive before. I started a RonDB cluster and wrote a client that could send and receive signals directly, bypassing the real NDB API. With some modifications to the debug build of RonDB, I had a working unit test framework. It took slightly longer than expected since Claude needed to learn a few things about writing this kind of test program — very few existing test programs did similar things.

Scaling Up with Parallel Sessions

After writing the first test case, I wanted three things: deeper test coverage, a performance benchmark, and support for aggregation with CASE statements (very common in real-world queries, but not yet supported in single-table aggregation). Each of these was a self-contained mini-project that needed a test program similar to the one I had already built.

I had learned that Claude spends significant time thinking, so to maximise productivity, I launched three parallel Claude sessions — one for each task. All three were completed within two to three hours, even though I started late in the evening. The next morning, I could build a real-world benchmark running TPC-H Q12.

Figure 3: Running three Claude sessions in parallel to maximise development throughput

Key Takeaways

1. Unit testing distributed systems is now feasible — even with limited budgets, Claude can generate the 3–5x test code volume needed alongside your implementation.
2. Divide your task into modules — start from the low-level parts and build upward. In this case, beginning with the local database layer worked best.
3. Architecture first, then implementation — for each module, start by asking Claude for an architecture plan, then an implementation plan.
4. Expect many iterations — plan for constant reviews of the code Claude produces, especially the performance-critical parts.
5. Start with a simple test, then expand — write a basic test program first, then use it as a template for comprehensive coverage, benchmarks, and edge cases.

Your New Role: Architect, Manager, and Performance Expert

Claude can be remarkably productive when used correctly, but your role as a programmer fundamentally changes. You become an architect and manager while simultaneously needing to understand code at the deepest level. Learning low-level performance characteristics is just as important as it ever was — the performance-critical parts must still be fully understood by the developer. Claude can assist, but you need to know how to direct it.

Let Claude handle what it does best: building hash tables, linked lists, and other data structures it probably understands better than most developers. Let Claude suggest approaches where you are not certain of the best path forward. But keep the architectural vision firmly in your own hands.

Teaching Claude About Your Codebase

Programming with Claude is teamwork where you are the director, but your assistant has deep knowledge in some areas and can quickly absorb new information. Sometimes, though, it needs your high-level understanding to truly grasp what is happening in the code. Do not expect the code itself to describe all the details — the architecture is often invisible when you dive into the implementation. If it is invisible to a human, it is likely invisible to Claude as well.

To manage the knowledge Claude builds, we developed a structure using a root CLAUDE.md file that indexes all the domain knowledge in a directory called claude_files, with one subdirectory for each area we have built knowledge about. This is an early approach to managing institutional knowledge for AI assistants, but it is an important consideration for any team adopting these tools.

---

This article was written by the author and refined with Claude.