One billion rows from an Excel model: the 1BRC with generated Rust

We pointed Codcel at the One Billion Row Challenge: a 12 GB CSV, an ordinary Excel workbook, and a generated Rust REST server. Here's how it works and what it measured.

The One Billion Row Challenge (1BRC), created by Gunnar Morling, asks how fast you can aggregate one billion temperature readings from a text file: the minimum, mean and maximum for each weather station. Each line looks like this:

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8

The top entries are hand-tuned Java written specifically for this file. We wanted to answer a different question: can a model built in plain Excel handle a dataset this size once Codcel turns it into code?

The setup

  • The 1BRC generator script produced a file with 1,000,000,000 rows, about 12 GB, which we saved as T_Measurements.csv.
  • The file was used as is: not sorted, not pre-aggregated, not cleaned.
  • An ordinary Excel workbook defined the calculation: for a given town, return the min, average and max.
  • The CSV was attached to the workbook as a massive table, Codcel's table type for data beyond Excel's 1,048,576-row limit.

We then generated a Rust REST server and ran it with cargo run --release.

How Codcel handles a table that size

Massive tables go through a pipeline instead of being loaded into memory:

  1. CSV import. The CSV is read during code generation.
  2. Parquet conversion. The data is converted to Apache Parquet, a columnar format suited to analytical queries.
  3. SQL query engine. At runtime, the generated code queries the Parquet files with Apache DataFusion.
  4. Caching. Results are cached for five minutes, and identical concurrent queries are coalesced into one execution.

The same lookup functions you use in the workbook (VLOOKUP, XLOOKUP, INDEX/MATCH, FILTER and others) work against massive tables.

The result

A single POST to the generated server:

POST http://localhost:3030/combined
Content-Type: application/json

{ "town": "Miami" }

returned:

{ "min": -25.7, "average": 24.9, "max": 75.3 }

On a MacBook Pro with an M1 Pro, the first request took 8 seconds. The same request again took 1.5 seconds, thanks to the five-minute cache. There are roughly 2.5 million readings per town, scattered randomly across the billion rows.

What this does and doesn't show

It does not compete with the fastest 1BRC entries, which process the whole file in a couple of seconds with code written for exactly this input. Codcel is generic: nothing was tuned for this dataset.

It does show that a model an analyst can build and check in Excel can be pointed at data far beyond what Excel can open, and served as an API, without anyone rewriting the logic. In practice you'd split data like this into per-entity tables, which would make it faster still.

The full walkthrough, including how to reproduce it, is in the docs: One Billion Row Challenge. You can also watch it on YouTube in I Gave Excel One Billion Rows… and It Built a Web App.

Curious about the Rust output? See Excel to Rust.

Keep reading