A year of RDF in DuckDB

I’ve long felt that one of the biggest reasons RDF hasn’t gained wide horizontal traction (beyond forced cases like schema.org) is the lack of tooling and support from major data platforms.

A data platform that has gained a lot of traction in the last couple of years is DuckDB. It’s not hard to understand why: it filled a gap that existed (open source OLAP) with a tool that is very user friendly, cross platform, opinionated (in the best way), meets users where they are (consider the CSV and Excel support) and, most of all, very high performance.

A year ago I started experimenting with manipulating RDF using DuckDB, abusing the CSV parser to deal with nTriples and nQuads. I was curious what the performance would be like: turned out it could beat things like HDT for certain workload.

Fast forward a year and Claude, Devin, Kiran Chandra and I have created the fully featured community extension for RDF in DuckDB, supporting read, write and, as of last weekend, a good portion of SPARQL 1.1 query. With ~1000 downloads a week it’s clearly providing value.

Thanks to GenAI I’ve gone from super rusty knowledge of C++, none of CMake, CTest or DuckDB internals to productive. (A call out as well to David Robilla’s high performance portable serd library which underpins the read and write support)

GenAI is the biggest lift here. Last August I hadn’t written production C or C++ for 28 years. With Chat GPT and Gemini I was coding, debugging build issues and understanding the DuckDB code base. By Q1 Claude had become the key tool and my role has become a mix of technical architect, quality manager and product owner.

Could anyone, even with no software engineering experience, have replicated this? Certainly not last year. This year the gap has closed, yet I still believe a computer science degree helps. Would they know to break the code base into a dedicated library for SPARQL and R2RML support? Would they understand the way that separation simplifies unit testing, increases coverage, permits support for other database engines? Would they be opinionated about how to build SPARQL support, what overly stretches the metaphor, what will be useful?

As a technology leader I’ve long been used to architecting systems, dividing up the work, thinking in terms of composable units and system contracts. GenAI means that I can implement that at pace with work/life balance.

It’s not clear to me where this ends, but as someone who always loved to create, I’m enjoying the ride.