We need a fast way to query multiple potentially large sets of data on-demand at interactive speeds. Sometimes the easiest solution to a hard problem is to build the right tool for the job.
Designing Data-Intensive Applications
Data is at the center of many challenges in system design today. Difficult issues need to be figured out, such as scalability, consistency, reliability, efficiency, and... - Selection from Designing Data-Intensive Applications [Book]
Dremel: interactive analysis of web-scale datasets: Proceedings of the VLDB Endowment: Vol 3, No 1-2
Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system ...
An entire Social Network in 1.6GB (GraphD Part 2)
Roaring Bitmaps offer an even more efficient way to store and query an entire social graph, fitting the entire network of 5.5M users and 164M+ follows into a ~1.6GB SQLite DB on disk.
surrealdb/surrealdb
A scalable, distributed, collaborative, document-graph database, for the realtime web
CompactLTJ: Space & Time Efficient Leapfrog Triejoin on Graph Databases
Leapfrog Triejoin (LTJ) is arguably the most practical and popular worst-case-optimal (wco) algorithm for solving basic graph patterns in graph databases. Its main drawback is that it needs the database triples (subject, predicate, object) represented as paths in a trie, for each of the six orders of subject, predicate, and object. The resulting blowup in space makes most systems disregard LTJ or implement it only partially, which makes their corresponding algorithms non-wco. In this paper we show that, by using compact data structures, it is possible to build an index that at the same time matches the query time performance of the fastest classic wco index, and uses a fraction of the space of non-wco indices (which are much slower). Concretely, we make use of compact tree representations to store functional tries using one bit per trie edge, instead of one pointer, and further reduce the space by storing partial tries. Our most compact variant uses 5–6 times less space than classic wco implementations and 2–3 times less than classic non-wco systems. At solving queries, it is on par with the fastest classic wco system, and 30–40 times faster than non-wco systems. We further incorporate improved query resolution strategies into CompactLTJ variants, which makes it considerably faster than classic wco systems as well, on queries that do not output too many results. Finally, we show how CompactLTJ can incorporate dynamism without altering its performance, even under very demanding update regimes. We leave a public fully-functional implementation of CompactLTJ that can be directly used by practitioners.
Parallel Web Systems | Infrastructure for intelligence on the web
Parallel's new FindAll API turns natural language queries into custom datasets from the web. It finds entities like companies, people, or locations based on your criteria, then enriches them with structured data—all with citations. FindAll Pro achieves 61% recall, 3x better than competitors.
Designing Data-Intensive Applications (DDIA) — an O’Reilly book by Martin Kleppmann (The Wild Boar Book)
NoSQL… Big Data… Scalability… CAP Theorem… Eventual Consistency… Sharding…
Semantic Data Modeling, Graph Query, and SQL, Together at Last?
Our teams advance the state of the art through research, systems engineering, and collaboration across Google.
Designing Data-Intensive Applications
Data is at the center of many challenges in system desi…
turbopuffer: fast search on object storage
Inaugural blog post about the development of turbopuffer, a search engine that uses object storage and SSD caching for cost-effective, low latency search. This post describes into the motivation behind its creation, its unique architecture, and how it significantly reduces costs for large-scale vector searches. Discover how turbopuffer is transforming search infrastructure for companies like Cursor and Suno, offering a scalable and reliable solution.
DSPy on Twitter / X
DSPy is the highest-bandwidth language to talk to computers in a just-precise-enough way.First 36 lines below:1. Take arbitrarily long content: `chunks`.2. Build a global Table of Contents.3. Distribute chunks into sections to be written.4. Recursively write each section. pic.twitter.com/7NN61isFLs— DSPy (@DSPyOSS) May 13, 2025
GraphQL
GraphQL is a query language designed to build client applications by providing an intuitive and flexible syntax and system for describing their data requirements and interactions.
How we made geo joins 400× faster with H3 indexes
Using H3 indexes to greatly speed up geospatial SQL Queries. Orders of magnitude improvements can be gained with the right data structures
How we made geo joins 400× faster with H3 indexes
Using H3 indexes to greatly speed up geospatial SQL Queries. Orders of magnitude improvements can be gained with the right data structures