Vortex is an extensible, state-of-the-art columnar file format, with associated tools for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.
Column Storage for the AI Era
In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thinking is that the context has changed so much that the design of yore (the previous decade) is not going to cut it moving forward. This seemed a bit intriguing to me, especially since the main contribution of Parquet has been to provide a standard for columnar storage. Parquet is not simply a file format. As an open source project hosted by the ASF, it acts as a consensus building machine for the industry. Creating six new formats is not going to help with interroperability. I spent some time to understand a bit better how things actually changed and how Parquet needs to adapt to meet the demands of this new era. In this post I’ll discuss my findings.
Parquet
Apache Parquet Documentation Releases Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.
Columnar Storage is Normalization
Something I didn't understand for a while is that the process of turning row-oriented data into column-oriented data isn't a totally bespoke, foreign concept...
shshemi/tabiew
A lightweight TUI application to view and query tabular data files, such as CSV, TSV, and parquet.
RAFS Filesystem | dragonflyoss/nydus | DeepWiki
RAFS (Registry Acceleration File System) is a read-only, chunk-deduplicated, on-demand-loading filesystem designed for container image serving. The filesystem implementation is located in the `rafs/`
Files As Metadata Format
File structures with simple contents can be used instead of YAML, TOML or JSON. More effectively!
Nothing new under the sun: everything is a file
The Unix revolution was built on a key principle: everything is a file. Now, with the rise of AI Agents, LLMs have access to half a century of file-based arcana. The result? Everything is becoming a file again.
What is SoaML? How to Draw SoaML Diagrams?
Quick SoaML tutorial for fast learning of SoaML. It's written to explain what SoaML is, how to use SoaML for SOA and how to draw the various SoaML diagrams.
A primer on Roaring bitmaps: what they are and how they work
A detailed primer on Roaring bitmaps explaining what they are, how they're different from traditional bitmaps, and how they work internally.
Engineering High-Performance Parsers with Data-Oriented Design
Notes from building Yuku: the AST is flat arrays of u32 indices instead of a pointer tree, and memory layout, allocation, strings, unicode, and serialization all follow from that one decision.
Pinata | Autonomous File Storage
Add file uploads and retrieval in minutes so you can focus on your app —because you’ve got better things to code than infrastructure.
mohsen1/yek
A fast Rust based tool to serialize text-based files in a repository or directory for LLM consumption
Web Files
Interoperable web-files and ejectable apps.
AT Codes
Códigos relacionados com ATProto, para facilitar a busca através de diversas fontes e repositórios diferentes. | Multiple sources of codes using atproto in many ways, so it's easier to browse and search. PLEASE, only use PERMALINK for anything hosted on Tangled (or I will have to replace it). Gist and Strings are okay too.
DASL: Web Tiles
Web Tiles
Nothing new under the sun: everything is a file
A Social Filesystem — overreacted
File Over App: A Philosophy for Digital Longevity
Ejectable Apps
pds.dad/influential-posters
margin.at/margin
upvote.at/trending.upvote.at
upvote.at/upbot.upvote.at
upvote.at/upvote.at
upvote.at/lexicons