An entire Social Network in 1.6GB (GraphD Part 2)

Social Network User Data Storage

Social Network User Data Storage

Your Data Fits in Memory (GraphD Part 1)

We need a fast way to query multiple potentially large sets of data on-demand at interactive speeds. Sometimes the easiest solution to a hard problem is to build the right tool for the job.

A primer on Roaring bitmaps: what they are and how they work

A detailed primer on Roaring bitmaps explaining what they are, how they're different from traditional bitmaps, and how they work internally.

Cloud Atlas

Interactive map of the Bluesky social graph, initially compiled by https://theo.io, turbocharged with goose powers.

surrealdb/surrealdb

A scalable, distributed, collaborative, document-graph database, for the realtime web

Bluesky now has more than 10M users | TechCrunch

Social network Bluesky said Monday that it now has more than 10 million users. This is largely thanks to the rapid growth of the network in the past few

A survey of community search over big graphs

With the rapid development of information technologies, various big graphs are prevalent in many real applications (e.g., social media and knowledge bases). An important component of these graphs is the network community. Essentially, a community is a group of vertices which are densely connected internally. Community retrieval can be used in many real applications, such as event organization, friend recommendation, and so on. Consequently, how to efficiently find high-quality communities from big graphs is an important research topic in the era of big data. Recently, a large group of research works, called community search, have been proposed. They aim to provide efficient solutions for searching high-quality communities from large networks in real time. Nevertheless, these works focus on different types of graphs and formulate communities in different manners, and thus, it is desirable to have a comprehensive review of these works. In this survey, we conduct a thorough review of existing community search works. Moreover, we analyze and compare the quality of communities under their models, and the performance of different solutions. Furthermore, we point out new research directions. This survey does not only help researchers to have better understanding of existing community search solutions, but also provides practitioners a better judgment on choosing the proper solutions.

AT Explore

Explore and discover AT Protocol data with our fast, user-friendly browser. Search posts, profiles, and feeds across the decentralized social web — including apps like Bluesky.

Better bitmap performance with Roaring bitmaps

Bitmap indexes are commonly used in databases and search engines. By exploiting bit-level parallelism, they can significantly accelerate queries. However, they can use much memory, and thus we might prefer compressed bitmap indexes. Following Oracle's lead, bitmaps are often compressed using run-length encoding (RLE). Building on prior work, we introduce the Roaring compressed bitmap format: it uses packed arrays for compression instead of RLE. We compare it to two high-performance RLE-based bitmap encoding techniques: WAH (Word Aligned Hybrid compression scheme) and Concise (Compressed `n' Composable Integer Set). On synthetic and real data, we find that Roaring bitmaps (1) often compress significantly better (e.g., 2 times) and (2) are faster than the compressed alternatives (up to 900 times faster for intersections). Our results challenge the view that RLE-based bitmap compression is best.

Tim Berners-Lee, Giant Global Graph (2007)

A new vision for the web where the users master their data and link them to everybody's benefit in a giant global graph.

Looking AT the Blue Skies of Bluesky

The pitfalls of centralized social networks, such as Facebook and Twitter/X, have led to concerns about control, transparency, and accountability. Decentralized social networks have emerged as a result with the goal of empowering users. These decentralized approaches come with their own trade-offs, and therefore multiple architectures exist. In this paper, we conduct the first large-scale analysis of Bluesky, a prominent decentralized microblogging platform. In contrast to alternative approaches (e.g. Mastodon), Bluesky decomposes and opens the key functions of the platform into subcomponents that can be provided by third party stakeholders. We collect a comprehensive dataset covering all the key elements of Bluesky, study user activity and assess the diversity of providers for each sub-components.

Designing Data-Intensive Applications (DDIA) — an O’Reilly book by Martin Kleppmann (The Wild Boar Book)

NoSQL… Big Data… Scalability… CAP Theorem… Eventual Consistency… Sharding…

Building Bluesky: a Distributed Social Network (Real-World Engineering Challenges)

Bluesky is built by around 10 engineers, and has amassed 5 million users since publicly launching in February this year. A deep dive into novel design decisions, moving off AWS, and more.