Files As Metadata Format

fx – a terminal JSON viewer & processor

A terminal viewer & processor for JSON, YAML, & TOML (TUI and CLI)

Lume, the static site generator for Deno

Support for multiple file formats like Markdown, YAML, JavaScript, TypeScript, JSX, Nunjucks etc.

DASL: MASL — Metadata for Arbitrary Structures & Links

MASL is a CBOR-based metadata system that is designed to work well with content-addressed and decentralised systems, to enable fully self-contained, self-certified content distribution.

mohsen1/yek

A fast Rust based tool to serialize text-based files in a repository or directory for LLM consumption

JuiceFS - Open Source Distributed POSIX File System for Cloud

JuiceFS is a cloud-based high-performance distributed file system, based on object storage and database to achieve data and metadata separation architecture, easily carrying tens of billions of file storage, to meet the random read performance requirements of big data, AI, autonomous driving, gene sequencing and other massive data.

Parquet

Apache Parquet Documentation Releases Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.

JSON Formatter & Validator

Format and validate JSON data so that it can easily be read by human beings.

DASL — Data-Addressed Structures & Links

A small set of simple, standard primitives to work on content-addressed data.

DASL — Data-Addressed Structures & Links

A small set of simple, standard primitives to work on content-addressed data.

Designing better file organization around tags, not hierarchies

Computer users organize their files into folders because that is the primary tool offered by operating systems. But applying this standard hierarchical model to my own files, I began to notice shortcomings of this paradigm over the years. At the same time, I used some other information systems not based on hierarchical path names, and they turned out to solve a number of problems. I propose a new way of organizing files based on tagging, and describe the features and consequences of this method in detail.

DASL: Content-Addressable aRchives (CAR)

The CAR format offers a serialized representation of set of content-addressed resources in one single concatenated stream, alongside a header that describes that content.

Research Object Crate (RO-Crate)

RO-Crate is a community effort to establish a lightweight approach to packaging research data with their metadata. It is based on schema.org annotations in JSON-LD, and aims to make best-practice in formal metadata description accessible and practical for use in a wider variety of situations, from an individual researcher working with a folder of data, to large data-intensive computational research environments.

Column Storage for the AI Era

In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thinking is that the context has changed so much that the design of yore (the previous decade) is not going to cut it moving forward. This seemed a bit intriguing to me, especially since the main contribution of Parquet has been to provide a standard for columnar storage. Parquet is not simply a file format. As an open source project hosted by the ASF, it acts as a consensus building machine for the industry. Creating six new formats is not going to help with interroperability. I spent some time to understand a bit better how things actually changed and how Parquet needs to adapt to meet the demands of this new era. In this post I’ll discuss my findings.

Fil One Object Storage

S3-compatible object storage on Filecoin with no egress fees and verifiable data integrity, proven daily.