magmalake is an experiment: can a modern data stack — Apache Iceberg and
everything underneath it — exist in Mojo without reaching back through Python
or the JVM, and what does it cost to find out? Twelve small tins, each
independently usable, each checked against the reference implementation of
the format it claims to speak.
from iceberg import Catalog
var table = Catalog.rest("https://polaris.example/api/catalog").load("db.events")
var rows = table.scan().filter('["=", "region", "eu"]').select(["id", "amount"]).to_table()
Metadata, manifests, scan planning, Parquet decode, deletes and deletion
vectors, S3 IO and the REST catalog — all of it Mojo.
The question behind magmalake is narrow and answerable: can a modern
data stack exist natively in Mojo — and what does it cost to build one?
Not a demo, not a binding, not a thin veneer over an existing C library, but the
real thing: an Apache Iceberg implementation with the file formats, the codecs,
the hashes and the object storage all written underneath it.
Before this, none of it existed. There was no Parquet page decode in Mojo, no
Avro, no zstd or snappy or lz4, no Roaring bitmaps, no SHA-256, and no table
format anywhere in the ecosystem. Every Mojo dataframe reached Parquet by calling
through to PyArrow. So the experiment had to start at the bottom and work up.
What came out is twelve tins. Some of the answers are encouraging — native
Parquet decode is faster than pyarrow on a single core. Some are not:
writes are slower, thread-pooled scans lose, and the language is still missing
pieces a data stack assumes. Both halves are on this page, because an experiment
you only report the wins from is not an experiment.
The stack
Twelve tins, one layering
Iceberg sits on Parquet, Avro, Roaring and objectstore; those sit on Thrift, the
compression codecs and the hashes; threads runs alongside all of it. Nothing is
bundled — take the one you need.
Every tin is Apache-2.0, published on mojoshelf, and CI-tested on stable Mojo 1.0.0
and the current nightly, across macOS and Linux. Each carries its own correctness
oracle — an outside implementation it has to agree with.
Native Apache Iceberg. Read a table's metadata.json, pick a snapshot, decode its manifests, plan a scan and read the rows — then create, append, delete, overwrite and expire. No JVM, no Python, no Rust in the path.
Format v1–v3 reads: position and equality deletes, v3 deletion vectors, schema evolution, nested types
The first native Apache Parquet reader and writer in Mojo. It decodes the footer, the page headers, the levels and the values itself, and hands the result back as Arrow arrays over the Arrow C Data Interface.
Every physical and logical type, every encoding, v1 and v2 pages
Nested lists, maps and structs reconstructed from definition and repetition levels
Row-group, page-index and bloom-filter pruning; field-id projection for Iceberg
Reads 1M rows in 4.3 ms — 232M rows/s, 1.9× pyarrow
Checked againstpyarrow, value-exact on 33 fixtures; pyarrow reads back every file it writes
Pure-Mojo Apache Avro: schema parsing, the binary encoding, Object Container Files both ways, and schema resolution. The core has no dependencies at all — not even FFI.
Storage and HTTP for Iceberg tables: a FileIO abstraction over local files, HTTP(S) range reads and S3, with the pooled HTTP transport the rest of the stack was missing.
S3 with SigV4, vended credentials, presigned URLs and multipart upload
Pooled libcurl transport — 0.15 ms per range read on a reused connection
Pure-Mojo SHA-256 and HMAC on hardware crypto paths, 2.7 GB/s
GCS, Azure and plain HTTP range reads alongside local files
Checked againstAWS SigV4 suite 37/37, and S3 verified end-to-end against MinIO in CI
Apache Thrift serialization in pure Mojo — compact and binary protocols — plus every struct, union and enum of the Parquet metadata schema, generated ahead of time.
TCompactProtocol and TBinaryProtocol behind one trait
Minimal OS threads for Mojo: spawn and join pthreads, share state through atomics and a mutex, and fan a loop out over cores with parallel_for. A stopgap, distilled from flare, until the language ships its own.
parallel_for over cores — Mojo currently ships no other way to use a second one
Atomics that bridge the stable/nightly std.atomic split
Mutex, spawn, join and thread pinning
Spawn and join in 14 µs; parallel_for scales ~4×
Checked againstContended-count and memory-visibility proofs designed to give a wrong number, not a flake
Apache Iceberg over a thin Rust cdylib wrapping iceberg-rust behind a C ABI. Superseded — no longer required for any operation — and kept only as a third independent implementation to check the native one against.
56 extern "C" functions over one shared tokio runtime
SQL/JDBC catalog against real object storage
Kept as a cross-implementation oracle, not a dependency
Checked againstPyIceberg reading through the same sqlite catalog file the binding wrote
Performance
Measured, including where it loses
Apple M4, single core unless a row says otherwise. Same machine, same files,
reproducible from each repository with pixi run bench. magmalake beats
pyarrow on single-core Parquet reads; it loses on writes and on multi-threaded scans,
where the remaining Iceberg gap is largely libzstd itself.
Table and file formats
Parquet decode is where the native stack pays off; writes and thread-pooled scans are where it does not, yet.
Operation
magmalake
Reference
Parquet read, 1M rowsint64 / double / dictionary
4.3 ms — 232M rows/s
pyarrow 8.2 ms — 1.9× faster
Parquet read, 4 coresthreads.parallel_for over row groups
660M rows/s
memory-bandwidth bound
Parquet write, 1M rows
42 ms
pyarrow 31 ms — slower
Parquet footer1,000 columns × 50 row groups
78 ms read / 8 ms write
—
Iceberg scan, 1M rowszstd-compressed
35.8 ms — 12.1 ms on 4 workers
pyarrow single-thread 26.8 ms on the same files; PyIceberg 7.9 ms using its thread pool
Iceberg append, 1M rowsdata files, manifests and commit
233 ms
PyIceberg ~150 ms
Iceberg scan planning, 500 manifests31.9 µs fixed cost per manifest, 11.5 µs of it the file-read floor
21.4 ms
62.3 ms before the schema and plan cache
Avro decode, manifest-shaped records
19.2M records/s — 27.5M with field selection
fastavro 1.74M — 11–13× faster
Avro inflate
860 MB/s
—
scroll the table sideways →
Primitives
Codecs, hashes and threads — the layer everything above is only as fast as.
Operation
magmalake
Reference
SHA-256pure Mojo with ARMv8 crypto / SHA-NI intrinsics
2.7 GB/s — 610 MB/s scalar fallback
OpenSSL 3.2 GB/s
zstd / lz4 decompressFFI
10–14 GB/s / 8–19 GB/s
—
snappy decompresspure Mojo
up to 20 GB/s incompressible, ~3 GB/s compressible
—
CRC-32 / murmur3 / XXH64pure Mojo
1.2–1.5 GB/s
—
threads: spawn and join
14 µs — parallel_for scales ~4×
memory-bandwidth ceiling
scroll the table sideways →
Object storage
Measured against MinIO on the same machine, so the network is not the story.
Operation
magmalake
Reference
S3 multipart upload, 16 MB
409 MB/s
53 MB/s when payload hashing was still scalar
HTTP range read, pooled connection
0.15 ms local / 0.52 ms signed S3
19× / 11× faster than a fresh connection per request
scroll the table sideways →
Threads, for now, come from a tin
Mojo's standard library currently has no threading. parallelize was
removed and there is no thread pool reachable from user code, so every number
above that mentions more than one core goes through
magmalake/threads,
which wraps pthreads directly via external_call.
That tin is inspired by and distilled from
flare
— Ehsan Mokhtarian's project, MIT-licensed — which worked out how to get real OS
threads out of Mojo in the first place. threads.mojo keeps the parts a data stack
needs and drops the rest.
It is explicitly a stopgap. When the language ships its own threading, this tin
should stop existing.
How these numbers were found
Four optimisation passes, each of which located its bottleneck by profiling rather
than by guessing. None of them was where it seemed obvious to look.
A dlopen on every single decompress callAbout 450 µs per call, spent opening a library that was already open.
SHA-256 running 45× off what the hardware can doThe scalar implementation was correct and slow; the CPU had crypto instructions sitting idle.
Avro boxing roughly 60 allocations per recordAnd JSON schema parsing turning out to be 72% of the cost of reading a manifest.
500 byte-identical manifest schemas, each parsed from scratchCaching the parsed schema and the plan cut scan planning from 62.3 ms to 21.4 ms.
Correctness
Gated on someone else's implementation
A format library that only passes its own tests has proved nothing. Every format tin
here is checked against the reference implementation of the format, in both
directions where the format has two.
Iceberg
Row-sets cell-exact against PyIceberg 0.11.1 and DuckDB 1.5.5, in
both directions, including deletes, deletion vectors and schema evolution. Tables
magmalake writes are read back cell-exact by both — and PyIceberg can append to a
table magmalake created.
Parquet
Value-exact against pyarrow on 33 fixtures — every value of every column — and
pyarrow reads back every file magmalake writes. The Arrow export imports through
pyarrow.Array._import_from_c.
Avro & Thrift
Avro round-trips against fastavro both ways across all four codecs. Thrift is
byte-identical to Apache Thrift on generated wire vectors, in
both the compact and the binary protocol.
S3 & SigV4
37 of 37 cases of the official AWS SigV4 test suite, every stage.
S3 is verified end-to-end against MinIO in CI, not only on a
developer's machine.
Roaring
Byte-exact against pyroaring in both directions, with pyroaring-produced payloads
baked into the tests as constants so the oracle cannot drift.
Everything else
Codecs against Python zstandard, lz4 and
python-snappy; hashes against zlib, mmh3 and xxhash plus the Iceberg
spec's own vectors; threads against contended-count and memory-visibility proofs.
Building it found bugs upstream
Holding three implementations against each other surfaces disagreements, and not all
of them were magmalake's fault. Each is documented in the repository that found it.
PyIcebergv3 manifest schemas mishandled on both the read and the write path
pyarrowstatistics under list<struct>, and codec and logical-type reporting
iceberg-rustmetrics pruning
Status
Active, v0.x
Reads are complete for Iceberg format v1 through v3. Writes cover the common table
operations. Several things are genuinely missing, and they are listed here rather
than left to be discovered.
Works today
Iceberg format v1–v3 reads, with position deletes, equality deletes and v3 deletion vectors
Schema evolution and nested types through the read path
Fast-append, row-level delete both copy-on-write and merge-on-read
Overwrite and dynamic partition overwrite
expire_snapshots
Filesystem and REST catalogs
Local, HTTP, S3, GCS and Azure IO
Not yet
Compaction — rewrite_manifests
Encryption
Brotli
Predicates inside list and map elements
The spec's v4 drafts — relative paths and content_stats — are tracked
but not implemented.
Take a tin
Each of the 12 repositories is independently usable and independently
installable. Start with whichever layer you need.