In the past few years, we’ve seen a Cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thinking is that the context has changed so much that the design of yore (the previous decade) is not going to cut it moving forward. This seemed a bit intriguing to me, especially since the main contribution of Parquet has been to provide a standard for columnar storage. Parquet is not simply a file format. As an open source project hosted by the ASF, it acts as a consensus building machine for the industry. Creating six new formats is not going to help with interoperability. I spent some time to understand a bit better how things actually changed and how Parquet needs to adapt to meet the demands of this new era. In this post I’ll discuss my findings.
Parquet
Apache Parquet Documentation Releases Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.
Vortex | An extensible, SOTA columnar file format
Vortex is an extensible, state-of-the-art columnar file format, with associated tools for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.
Columnar Storage is Normalization
Something I didn't understand for a while is that the process of turning row-oriented data into column-oriented data isn't a totally bespoke, foreign concept...
JuiceFS - Open Source Distributed POSIX File System for Cloud
JuiceFS is a cloud-based high-performance distributed file system, based on object storage and database to achieve data and metadata separation architecture, easily carrying tens of billions of file storage, to meet the random read performance requirements of big data, AI, autonomous driving, gene sequencing and other massive data.
Why Observability Requires a Distributed Column Store
Alex explains distributed column stores, how they work, why they're so fast, and why that's a fundamental requirement for observability.
Fil One Object Storage
S3-compatible object storage on Filecoin with no egress fees and verifiable data integrity, proven daily.
Nothing new under the sun: everything is a file
The Unix revolution was built on a key principle: everything is a file. Now, with the rise of AI Agents, LLMs have access to half a century of file-based arcana. The result? Everything is becoming a file again.
shshemi/tabiew
A lightweight TUI application to view and query tabular data files, such as CSV, TSV, and parquet.
From Albums to Streams: How Modularity Changes Systems
Scientific publishing is breaking under document-centric formats designed for a physical world. Borrowing from music’s shift from albums to streaming, we make the case that open access alone cannot deliver reuse, trust, or scale. The future of science depends on modular, interoperable research components that move across tools, enabling new workflows, tools, and ecosystems.
DuckLake Architecture Deep Dive
DuckLake is a next generation data lakehouse and open table format. It is significantly simpler and faster than Apache Iceberg or Delta Lake because it uses a SQL database for storing the catalog and metadata. This post covers an architecture deep dive with a detailed Q&A section. DuckLake consists of 3 components: storage, catalog, and compute. Each can be scaled independently, hosted locally with ease, and deployed to the cloud for production. MotherDuck offers managed DuckLake lakehouses for additional simplicity and performance.
Engineering High-Performance Parsers with Data-Oriented Design
Notes from building Yuku: the AST is flat arrays of u32 indices instead of a pointer tree, and memory layout, allocation, strings, unicode, and serialization all follow from that one decision.
Ryan Dahl on Twitter / X
Introducing celld: a self-hosted, distributed Durable Objects and Workers implementation- celld = V8 + S3 + SQLite + LTX+ Tokio- programmed with exactly the Cloudflare Workers/DO JavaScript APIs and config- order of magnitude cheaper at scale- writes are durable before… pic.twitter.com/KCBf6U5DQw— Ryan Dahl (@rough__sea) August 5, 2026
This is a radically different idea than the advocacy I’ve seen for “data portability” and ”data sovereignty”. It’s a *restructuring* of power rather than small adjustments to existing structures. atproto and local-first, both in their own ways, demonstrate what such a restructuring can look like.
Orion Reed
We need to structurally separate user data from applications so that applications become *users* of data rather than *containers* for it.
I was looking at an older version of the software I'm yet again working on from what, 15 years ago, and I can't believe we're still mostly stuck with the same private blob storage options that can serve as user-data storage. S3-compatible storage + Dropbox was the list: github.com/icidasset/ongaku-ryoho-v1?tab…
