Parquet

Column Storage for the AI Era

In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thinking is that the context has changed so much that the design of yore (the previous decade) is not going to cut it moving forward. This seemed a bit intriguing to me, especially since the main contribution of Parquet has been to provide a standard for columnar storage. Parquet is not simply a file format. As an open source project hosted by the ASF, it acts as a consensus building machine for the industry. Creating six new formats is not going to help with interroperability. I spent some time to understand a bit better how things actually changed and how Parquet needs to adapt to meet the demands of this new era. In this post I’ll discuss my findings.

shshemi/tabiew

A lightweight TUI application to view and query tabular data files, such as CSV, TSV, and parquet.

JuiceFS - Open Source Distributed POSIX File System for Cloud

JuiceFS is a cloud-based high-performance distributed file system, based on object storage and database to achieve data and metadata separation architecture, easily carrying tens of billions of file storage, to meet the random read performance requirements of big data, AI, autonomous driving, gene sequencing and other massive data.

Vortex | An extensible, SOTA columnar file format

Vortex is an extensible, state-of-the-art columnar file format, with associated tools for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.

Files As Metadata Format

File structures with simple contents can be used instead of YAML, TOML or JSON. More effectively!

TigerFS

A filesystem backed by PostgreSQL, and a filesystem interface to PostgreSQL. TigerFS mounts a database as a directory. Every file is a real row. Writes are transactions. Multiple agents and humans can read and write concurrently with full ACID guarantees, locally or across machines. Any tool that works with files works out of the box.

DuckLake Architecture Deep Dive

DuckLake is a next generation data lakehouse and open table format. It is significantly simpler and faster than Apache Iceberg or Delta Lake because it uses a SQL database for storing the catalog and metadata. This post covers an architecture deep dive with a detailed Q&A section. DuckLake consists of 3 components: storage, catalog, and compute. Each can be scaled independently, hosted locally with ease, and deployed to the cloud for production. MotherDuck offers managed DuckLake lakehouses for additional simplicity and performance.

The HDF5® Library & File Format - The HDF Group - ensuring long-term access and usability of HDF data and supporting users of HDF technologies

HDF5® Technical Details License: BSD-style Official Media Type: application/vnd.hdfgroup.hdf5 Standard File Extension: .h5, .hdf5 High-performance data management and storage suite Utilize the HDF5 high performance data software library and file format to manage, process, and store your heterogeneous data. HDF5 is built for fast I/O processing and storage. Download HDF5 Documentation What is HDF5®? HETEROGENEOUS DATA HDF® sup ports […]

Linked Open Vocabularies (LOV)

Your entry point to high quality and reusable Vocabularies to describe Linked Data.

Frontend-only Multi-Player. Unlimited Bandwidth. Or: What is Croquet.io, really?

A multi-player web app needs a backend, right? What if I told you, it doesn’t? Read on for how Croquet gets rid of servers running your mult...

bigmoves/quickslice

Auto-indexing service and GraphQL API for AT Protocol Records

Documentation

Introduction Apache Avro™ is a data serialization system. Avro provides: Rich data structures. A compact, fast, binary data format. A container file, to store persistent data. Remote procedure call (RPC). Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages. Schemas Avro relies on schemas.

Apache Calcite | Proceedings of the 2018 International Conference on Management of Data

As part of the Digital Library's transition to Open Access, new features for researchers are available in the Premium Edition. Click here to learn more.

WDC - RDFa, Microdata, and Microformat Data Sets

More and more websites have started to embed structured data describing products, people, organizations, places, and events into their HTML pages using markup standards such as Microdata, JSON-LD, RDFa, and Microformats. The Web Data Commons project extracts this data from several billion web pages. So far the project provides 12 different data set releases extracted from the Common Crawls 2010 to 2023. The project provides the extracted data for download and publishes statistics about the deployment of the different formats.

Specification of graph translators with triple graph grammars

Data integration is a key issue for any integrated set of software tools. A typical CASE environment, for instance, offers tools for the manipulation of requirements and software design documents, and it provides more or less sophisticated assistance for keeping these documents in a consistent state. Up to now, almost all data consistency observing or preserving integration tools are hand-crafted due to the lack of generic implementation frameworks and the absence of adequate specification formalisms. Triple graph grammars are intended to fill this gap and to support the specification of interdependencies between graph-like data structures on a very high level. Furthermore, they are the fundamentals of a new machinery for the production of batch-oriented as well as incrementally working data integration tools.

GitHub - clarisma/geodesk: Fast and storage-efficient spatial database engine for OpenStreetMap data

Fast and storage-efficient spatial database engine for OpenStreetMap data - clarisma/geodesk