Add OPFS support by dengkunli · Pull Request #1490 · duckdb/duckdb-wasm

This PR adds OPFS (Origin Private File System) support for duckdb-wasm. With OPFS support, we could persist the database in the browser 🔥 API To remain simple and intuitive, no new api is added, ju...

DuckLake Architecture Deep Dive

DuckLake is a next generation data lakehouse and open table format. It is significantly simpler and faster than Apache Iceberg or Delta Lake because it uses a SQL database for storing the catalog and metadata. This post covers an architecture deep dive with a detailed Q&A section. DuckLake consists of 3 components: storage, catalog, and compute. Each can be scaled independently, hosted locally with ease, and deployed to the cloud for production. MotherDuck offers managed DuckLake lakehouses for additional simplicity and performance.

Data-at-Rest Encryption in DuckDB

DuckDB v1.4 ships database encryption capabilities. In this blog post, we dive into the implementation details of the encryption, show how to use it and demonstrate its performance implications.

Column Storage for the AI Era

In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thinking is that the context has changed so much that the design of yore (the previous decade) is not going to cut it moving forward. This seemed a bit intriguing to me, especially since the main contribution of Parquet has been to provide a standard for columnar storage. Parquet is not simply a file format. As an open source project hosted by the ASF, it acts as a consensus building machine for the industry. Creating six new formats is not going to help with interroperability. I spent some time to understand a bit better how things actually changed and how Parquet needs to adapt to meet the demands of this new era. In this post I’ll discuss my findings.

Blob store design challenges

It's just hashed data, what's so complex about a blob store anyway?

Parquet

Apache Parquet Documentation Releases Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.

Rethinking Digital Storage for Community-Based Archives

Over the last few decades, the standard procedure for storing and backing up data has been ‘put it in the cloud.’ Many of us use cloud…

JuiceFS - Open Source Distributed POSIX File System for Cloud

JuiceFS is a cloud-based high-performance distributed file system, based on object storage and database to achieve data and metadata separation architecture, easily carrying tens of billions of file storage, to meet the random read performance requirements of big data, AI, autonomous driving, gene sequencing and other massive data.

The Magic of Small Databases

Notes on personal libraries, collections and small indexes on the web

The Comprehensive Digital Preservation Platform

Create, curate, and collaborate on living archives. Start with a FREE GB.

file collection

A file collection is a set of computer files that is deliberately maintained by someone. In today's computing, most non-personal files can be considered temporary: they have been downloaded from somewhere and can always be redownloaded if needed again. The practice of actual file collection is becoming ever more marginal, but it has a place in permacomputing for increasing resilience and reducing network dependency.

Nothing new under the sun: everything is a file

The Unix revolution was built on a key principle: everything is a file. Now, with the rise of AI Agents, LLMs have access to half a century of file-based arcana. The result? Everything is becoming a file again.

DuckDuckGo - Protection. Privacy. Peace of mind.

The Internet privacy company that empowers you to seamlessly take control of your personal information online, without any tradeoffs.

jcsalterego/duckdb-atproto

Contribute to jcsalterego/duckdb-atproto development by creating an account on GitHub.