Columnar Storage is Normalization

Column Storage for the AI Era

In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thinking is that the context has changed so much that the design of yore (the previous decade) is not going to cut it moving forward. This seemed a bit intriguing to me, especially since the main contribution of Parquet has been to provide a standard for columnar storage. Parquet is not simply a file format. As an open source project hosted by the ASF, it acts as a consensus building machine for the industry. Creating six new formats is not going to help with interroperability. I spent some time to understand a bit better how things actually changed and how Parquet needs to adapt to meet the demands of this new era. In this post I’ll discuss my findings.

Why Observability Requires a Distributed Column Store

Alex explains distributed column stores, how they work, why they're so fast, and why that's a fundamental requirement for observability.

Vortex | An extensible, SOTA columnar file format

Vortex is an extensible, state-of-the-art columnar file format, with associated tools for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.

Personal data storage is an idea whose time has come

Data Ownership as a conversation changes when data resides primarily with people-governed institutions rather than corporations.

What makes something data?

This is a question I posted on BlueSky on Friday 11/21/25, inspired by a talk I recently attended about evaluation of “AI” systems. I think…

Random Tech Thoughts

This article is about standardisation, particularly how it relates to understanding code.  I’ll first go into an historical example that highlights the lack of standardisation in an area where we take it for granted today.  After that I’ll get into code, and how understanding code is like understanding data via visualisations.  In both the historic … Continue reading Standardisation and code

The Brick Protocol

When storage cost money per kilobyte, when disks were finite, when networks were slow—we needed containers. We needed to know where things were. The file system was a map of a small territory.

Parquet

Apache Parquet Documentation Releases Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.

dbt-labs/dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

DASL — Data-Addressed Structures & Links

A small set of simple, standard primitives to work on content-addressed data.

DASL — Data-Addressed Structures & Links

A small set of simple, standard primitives to work on content-addressed data.

Data Supply Chains

Data is a critical resource. Like oil, gold, or lithium, both companies and countries covet data. Ultimately, like oil, data’s flow can enrich those that posses

Against SQL

But SQL is the only widely-used implementation of the relational model, and it is:

Research Object Crate (RO-Crate)

RO-Crate is a community effort to establish a lightweight approach to packaging research data with their metadata. It is based on schema.org annotations in JSON-LD, and aims to make best-practice in formal metadata description accessible and practical for use in a wider variety of situations, from an individual researcher working with a folder of data, to large data-intensive computational research environments.