Experiences from the new wave of AI programming

How we built RonSQL in RonDB with Claude and Codex

A false start

A few years ago at Hopsworks we made an early attempt at using ChatGPT for coding. It turned out to be an expensive mistake: the generated code cost us several months of rewriting. So when I came back to AI programming this year, I did so with a fair amount of skepticism.

Trying again — the small wins

In January we tried again, and the first success came from an unexpected place. Our marketing manager put together a simple RonDB client in about two hours. That caught my interest. I took the code over and, in a few more hours, had extended it significantly.

Encouraged, I tried something closer to real engineering: extending our REST API server so that it could handle not only batched key reads, but also batched key inserts, writes, and deletes. The original code had taken significant effort to write; the extension took just two days. Clearly, extending existing code was something Claude and Codex could do well.

The big bet: a ten-year TODO item

With those successful experiments behind me, I realised this new tooling might be extremely useful for developing genuinely new features in RonDB.

Something our customers have wanted for a long time is more complex queries in real-time AI inferencing. Supporting that requires CTEs and parallel join aggregation — a capability that has sat on the RonDB TODO list for more than ten years. It is very complex development work; with conventional coding I would estimate it as a two-year project, at least.

What this really amounts to is adding subsets of complex SQL support to a key-value store. RonDB sits in the same category as key-value stores such as Redis and DynamoDB, which are traditionally used for exactly this kind of real-time, low-latency serving — but which have no support for complex SQL queries at all. Bringing CTEs and parallel join aggregation into that world is what makes the effort both unusual and worthwhile.

Would it be feasible now, with AI programming? I decided to find out. This post shares what I learned over the past five months.

The short version: today we released RonDB 26.04.1, which ships beta support for RonSQL with CTEs and pushdown join aggregation.

The development model

From the start it was clear that the proven development model still applied. You begin with a high-level plan, move to a detailed implementation plan, and only then implement it step by step.

We went through this loop several times. Each plan had at least 10–20 phases, and sometimes many more.

Claude vs. Codex

Working with both models, my conclusion is that their strengths are complementary. Claude is better at keeping track of plans and remaining work. Codex is often better at solving hard problems, but it is sloppier about maintaining the plan and can lose track of what to do next. So for the most part I let Claude own plan maintenance and the less complex tasks, and handed the hard bugs to Codex.

Massive testing across four layers

One thing that struck me immediately is that AI programming makes it possible to generate massive amounts of unit tests — even for the distributed cases that are normally painful to test.

RonSQL query execution spans four layers in RonDB:

• LDM (Local Data Manager) — the lowest layer, six modules that handle local query execution, local recovery, and local checkpointing.
• TC (Transaction Coordinator) — coordinates join execution for complex queries. Partial results flow from the first tables in the join to the later ones. The TC drives execution from a Query Tree: essentially a program that specifies how to run the query, including interpreted code to be sent down to the LDMs.
• NDB API — defines the Query Tree through an API. Applications can use it directly, though that is uncommon because it is low-level.
• RonSQL — the top layer, which translates SQL into NDB API calls. The NDB API builds a Query Tree and sends it to the TC; the TC distributes execution across the RonDB nodes, and each part runs on the LDM in every node.

Normally you write test programs in RonSQL, because writing and maintaining programs against the LDM, TC, and NDB API layers by hand is very hard. AI programming changes that. It became feasible to develop and test the LDM layer first, before touching the TC layer, and the TC layer before touching the NDB API layer. This made it much easier to reach a working implementation across all four layers quickly.

The asynchronous challenge

AI programming works very well on the kind of sequential code the models have seen in training. But RonDB uses an asynchronous programming model, and here the models sometimes needed firm guidance to understand how things actually work. Things that were obvious to me were sometimes completely invisible to the model — but it was usually easy enough to teach it through the prompts. The reverse was true just as often: the models generate new code far faster than a human can think.

How do you verify the AI got it right?

So how do you verify that the generated code is correct? Here is the model I try to follow.

Phase 1 — Build to the plan

Follow the high-level plan, then the implementation plan, then implement step by step. In this phase you are the architect: you tell the model what to do at a high level, and for the most part you act as an operator who accepts its suggestions. I found it essential not to let the model change any code without stating exactly what it changed and how. With Claude this was the default behaviour; Codex had to be kept on a tight leash to stop it running ahead and generating code without checking whether it was correct. Seeing every diff gives you at least a working idea of what the new code does — and this is the phase to generate lots of tests.

Crucially, each step includes writing at least one test case and verifying that the step has been successfully executed before moving on to the next one. A step is not “done” until there is a passing test that proves it. This keeps the implementation grounded and stops small mistakes from compounding across the many phases of a plan.

Phase 2 — Review

Once the steps are done, review all the new code. I focus mostly on the code that affects actual execution, and only glance at test cases and plans. I used the model to generate documentation — with images and signalling diagrams — to better understand the new code. This phase is where you ensure the model follows RonDB's memory model and correctly handles node failures and other failures in query execution. The models also tend to generate a lot of duplicated code, so removing duplicated code paths is an essential step — some of these removed thousands of lines of production code that would otherwise have had to be maintained.

Phase 3 — Stress the feature with tests

We then wrote CTE test cases without regard to whether they were supported. When a feature was not supported, we either fixed it or tagged it as unsupported.

What remains is to minimise query latency and to keep reviewing the generated code.

Working in parallel

One final observation: while the model is thinking, there is plenty of time to work on something else. Most of the time I had two to four projects running in parallel. That is how, alongside RonSQL, prototypes appeared for a JIT compiler for the RonDB interpreters, for using fibers in RonDB, and for Deadlock Discovery. Only Deadlock Discovery is in the source tree so far, and it is disabled by default. All three are still work in progress.

Takeaway

Looking back, a feature that had been on the TODO list for over a decade — and that I would have scoped as a two-year effort — reached beta in five months. The work did not disappear; it changed shape. My role shifted from writing the code to architecting it, guiding the models, reviewing the result, and above all testing it. That, more than raw code generation, is where the new wave of AI programming earned its place in how we build RonDB.

RonSQL: a new SQL engine for RonDB with predictable low latency and CTEs

Today we released RonDB 26.04.1, a beta release. It contains a lot of new features, but the most interesting one is that RonSQL now supports pushdown join aggregation and CTEs, so that complex queries run with low, predictable latency.

RonDB has always been able to answer complex queries through a MySQL Server. The problem with that path is predictability. The application asks for an answer, but it has no guarantee about how fast that answer arrives: the MySQL optimizer picks a plan that may or may not parallelise the query across the RonDB data nodes, and a plan that looks fine on a small table can fall off a cliff as the data grows.

RonSQL takes a different contract. The rule is simple:

Anything RonSQL accepts can be pushed down to the RonDB data nodes for parallel execution.

If a query parses and plans in RonSQL, it runs as a parallel pushdown — there is no fallback to a slow, single-threaded plan. That means the latency of a complex query is something the application can actually reason about up front, instead of discovering it in production.

Why this matters: Feature Stores

RonSQL grew out of the needs of AI applications built on Feature Stores, and in particular on-demand (real-time) transformations in Hopsworks.

Traditionally an online Feature Store only does primary-key lookups. To keep those lookups fast, every feature has to be pre-computed and written back before serving. That works, but it has two costs:

1. Stale features. A feature is only as fresh as the last batch job that recomputed it. The events from the last few seconds — often the most predictive ones for fraud, recommendations, or anomaly detection — are not yet reflected.
2. Expensive BLOB packing. A common trick is to pack a range of values into a single BLOB (often Avro-encoded) so a whole feature group can be fetched in one lookup. But every change to any value means re-encoding the whole BLOB, which may hold hundreds of values.

RonSQL attacks both problems:

• Fresh features on the fly. Instead of pre-computing aggregations, you stream raw rows into RonDB and let RonSQL aggregate them at query time. A row inserted a second ago is included immediately, so the feature reflects what just happened.
• Index scans instead of BLOBs. Instead of packing values into a BLOB and re-encoding on every change, you store the values as ordinary rows and let RonSQL read them with an index scan. Updates become simple inserts and deletes — and deletes are usually handled for you by RonDB's row-level TTL, so old data ages out without any application code.

CTEs (Common Table Expressions, the SQL WITH clause) are what let you combine these two ideas in a single, readable query: aggregate the fresh fact rows in a CTE, then join the result against your normalised dimension tables.

A worked example: real-time card-fraud features

Consider a fraud-scoring model. At inference time it needs a feature vector for one card, computed over that card's most recent activity. The raw transactions arrive continuously and are inserted straight into RonDB:

-- Fact table: one row per card transaction, inserted in real time.
CREATE TABLE txn (
  txn_id       BIGINT       NOT NULL,
  cc_num       BIGINT       NOT NULL,   -- card / account identifier
  merchantkey  INT          NOT NULL,   -- references merchant.m_merchantkey
  amount       INT          NOT NULL,   -- minor units (cents)
  txn_time     DATETIME(6)  NOT NULL,
  is_declined  TINYINT      NOT NULL,
  PRIMARY KEY USING HASH (txn_id),
  -- Ordered index: range-scan one card's recent activity cheaply.
  INDEX idx_card_time (cc_num, txn_time)
) ENGINE=NDB
  COMMENT='NDB_TABLE=TTL=604800@txn_time';  -- auto-expire rows after 7 days

-- Small dimension table: replaces a per-card Avro BLOB of merchant attributes.
CREATE TABLE merchant (
  m_merchantkey INT          NOT NULL,
  m_category    VARCHAR(16)  NOT NULL,
  m_risk_score  INT          NOT NULL,
  PRIMARY KEY USING HASH (m_merchantkey)
) ENGINE=NDB;

Step 1 — a fresh feature vector with a single scan

The simplest on-demand feature is a scalar aggregate over the card's last hour of transactions. No pre-computation, no BLOB — just an index range scan that includes whatever was inserted milliseconds ago:

SELECT
  COUNT(*)                                          AS txns_1h,
  SUM(amount)                                       AS amount_1h,
  MAX(amount)                                       AS max_amount_1h,
  AVG(amount)                                       AS avg_amount_1h,
  SUM(CASE WHEN is_declined = 1 THEN 1 ELSE 0 END)  AS declines_1h
FROM txn
WHERE cc_num = 4716253018273645
  AND txn_time >= DATE_SUB('2026-06-29 14:30:00', INTERVAL 1 HOUR);

RonSQL turns the WHERE into an ordered-index range scan on idx_card_time — it touches only this card's last hour — and pushes the COUNT/SUM/MAX/AVG and the CASE expression down to the data nodes, which aggregate in parallel and return a single row.

Step 2 — combining fresh aggregation with a dimension join, using a CTE

Now suppose the model wants spend broken down by merchant category. The category does not live on the transaction — it lives on the merchant dimension. The classic Feature Store approach would denormalise the category into a packed BLOB per card. With RonSQL we keep the data normalised and join at query time:

WITH spend_by_merchant AS (
  SELECT merchantkey AS m,
         SUM(amount) AS spend,
         COUNT(*)    AS txns
  FROM txn
  WHERE cc_num = 4716253018273645
    AND txn_time >= DATE_SUB('2026-06-29 14:30:00', INTERVAL 1 HOUR)
  GROUP BY merchantkey
)
SELECT m.m_category                  AS category,
       SUM(spend_by_merchant.spend)  AS spend_last_hour,
       SUM(spend_by_merchant.txns)   AS txns_last_hour
FROM merchant AS m
JOIN spend_by_merchant ON spend_by_merchant.m = m.m_merchantkey
GROUP BY m.m_category;

This query is easy to reason about, top to bottom:

1. The CTE spend_by_merchant runs an ordered-index range scan on idx_card_time, restricted to one card over the last hour — the only large table in play. The data nodes aggregate SUM(amount) and COUNT(*) grouped by merchantkey, returning just a handful of rows (one per merchant the card touched in that hour).
2. The join attaches the merchant attributes. m_merchantkey is the primary key of merchant, so each row is resolved with a cheap primary-key lookup rather than another scan. merchant is a small dimension table.
3. The outer query re-aggregates the joined result by m_category, producing one row per merchant category — a compact, model-ready feature vector.

Every stage is a pushdown, and stages such as the index scan and the lookups run in parallel across the data nodes. We could even execute several CTEs in parallel. Because RonSQL guarantees the whole thing pushes down, the latency is bounded and predictable — which is exactly the contract a real-time inference path needs.

Running a RonSQL query

RonSQL is reachable two ways:

• REST (RDRS). The RonDB REST server exposes a RonSQL endpoint, which is the path used by online serving. It even keeps a built-in latency histogram so you can watch the predictable-latency promise hold in production. The rondb-cli shell sends a line straight to it with the RONSQL prefix.
• ronsql_cli. A standalone client for scripting and experimentation. It reads a query from --execute, --execute-file, or stdin and can emit results as JSON (ideal for a feature vector) or TEXT.

Both paths support EXPLAIN. Prefixing a query with EXPLAIN shows the chosen pushdown plan — which index drives each scan, which joins become lookups, and where the aggregation happens — so “will this be fast?” is a question you answer before you ship, not after.

What RonSQL supports today

RonSQL is a read-only, aggregation-focused SQL subset designed so that everything it accepts can be pushed down:

• Statements: SELECT only (plus EXPLAIN). No DDL/DML.
• CTEs: multiple, comma-separated WITH clauses (non-recursive); a CTE can be joined as a child or used as the driving table.
• Joins: INNER JOIN, LEFT [OUTER] JOIN, self-joins, and comma cross-joins over scalar CTEs. Equi-join conditions, including composite keys (a.x = b.x AND a.y = b.y).
• Filtering: rich WHERE — = <> < <= > >=, LIKE, IN (list), IS [NOT] NULL, AND/OR/XOR/NOT, arithmetic, bitwise ops, and CASE WHEN.
• Subqueries: EXISTS, IN (subquery), and scalar subqueries.
• Aggregates: COUNT(*), COUNT(expr), SUM, MIN, MAX, AVG.
• Grouping & shaping: GROUP BY (multi-column, any table), HAVING, ORDER BY ASC/DESC, LIMIT.
• Expressions: arithmetic, CASE WHEN, GREATEST/LEAST, and date/time functions DATE_ADD, DATE_SUB, EXTRACT, INTERVAL.
• Index hints: FORCE INDEX, USE INDEX, IGNORE INDEX.

Why express features in SQL at all?

Because the Feature Store has to compute the same feature in two very different settings. Batch training and batch inference run on engines like Spark SQL and DuckDB — both batch query engines, chosen for different characteristics (Spark scales the work across a cluster for very large datasets; DuckDB runs embedded and is hard to beat on a single node for moderate data). Online serving runs on RonSQL, computing the feature fresh at inference time. When all of them speak SQL, the same feature logic can be expressed as the same query text on each engine, which eliminates a notorious source of training/serving skew — features that subtly differ between the model's training data and what it sees live at inference.

Where RonSQL goes next

RonSQL is already useful, but there is a clear roadmap, much of it driven directly by Feature Store needs:

• Distinct-count features. COUNT(DISTINCT ...), and an approximate variant (HyperLogLog), to answer “how many distinct merchants / devices / countries in the last hour?” — a staple fraud signal. DISTINCT and OFFSET more generally.
• More aggregate functions. STDDEV and VARIANCE (for z-score features), and GROUP_CONCAT.
• Point-in-time correctness (“time travel”). As-of joins so the same RonSQL query can reconstruct a feature's value at a historical timestamp for training, exactly matching what online serving would have returned — closing the skew gap completely.
• Richer query shapes. Derived tables / subqueries in FROM, UNION, RIGHT/FULL OUTER JOIN, and recursive CTEs for hierarchy/graph features.
• Vector / embedding pushdown. Top-K nearest-neighbour search pushed to the data nodes, as embeddings increasingly live alongside scalar features.
• Cost-based join ordering. The planner currently joins left-to-right; reordering based on table/index statistics would make more queries fast by default.
• Continuous / materialised features. Incrementally maintaining a CTE's result as new rows arrive, blurring the line between on-demand and pre-computed features.

The core contribution stays the same: predictable low latency for complex queries over fresh data, expressed in portable SQL — exactly what an online Feature Store needs to serve fresh, skew-free features to an AI model.

RonDB gets a RDMA transporter supporting pNFS Lattice

Today, PEAK-AIO announced pNFS Lattice, a metadata server for Parallel NFS. It is running on top of RonDB using an RDMA transporter.

Now you might wonder, does RonDB have an RDMA transporter? A few weeks ago the answer would have been no. It has been on the TODO list for more than a decade, but it was not prioritized. However, a few weeks ago Eyal Lemberger from PEAK-AIO contributed an RDMA transporter to RonDB. Me and Eyal worked on the contribution for a few weeks and it was stable enough to integrate into RonDB 26.02, our latest stable release.

In the normal builds, the RDMA transporter is disabled, so it requires manual building to integrate it. The aim is to integrate the RDMA transporter into the RonDB release planned for the end of the year 2026.

3x Latency Improvement

RDMA improved the latency of metadata operations by a factor of 3. Actually, this brings RonDB back to its roots. The very first transporter that NDB Cluster (RonDB is a fork of NDB Cluster) had was a Dolphin SCI transporter. I have worked with Dolphin for more than 30 years. Today Dolphin technology can be used with a normal TCP/IP transporter using normal sockets.

The Wakeup Cost Challenge

The biggest challenge of an RDMA transporter is the same as the challenge of the shared memory transporter. It is about how to wake up the receiving process—it is important not to pay the wakeup cost for every signal.

In RDMA, one can send batches and the wakeup is handled by the network card. Thus, the sender has no cost attached to sending other than writing to the receivers memory. The wakeup cost is only at the receiving side and is integrated in the network card, so the only cost is an occasional interrupt to ensure that the receiver is awake.

Great to see RonDB coming to good use and the open-source license making the technology available to many. Many thanks to PEAK-AIO for the contribution.