When workflow metadata and application data live in the same Postgres database, they can be updated in the same database transaction, which simplifies tough problems like workflow task idempotency and atomicity.
DBOS - Durable Workflow Orchestration
DBOS is an open source durable execution and workflow orchestration system that radically simplifies the development and operation of reliable, observable workflows.
PowerSync: Backend DB - SQLite sync engine | For Postgres, MongoDB, MySQL, SQL Server
Automatically sync your backend database with in-app SQLite and avoid the complexities of using APIs to move app state over the network.
Data exchange: getting to the core | ACM Transactions on Database Systems
Data exchange is the problem of taking data structured under a source schema and creating an instance of a target schema that reflects the source data as accurately as possible. Given a source instance, there may be many solutions to the data exchange ...
FlowLog - Efficient and Extensible Datalog | FlowLog
FlowLog: Efficient and Extensible Datalog via Incrementality
Introducing Tap: Repository Synchronization Made Simple | Bluesky
Just about every app built on AT needs data from a repository at some point. For many use cases – feed generators, labelers, bots – streaming live data through a Relay or Jetstream works well. But some applications need to go beyond what Jetstream was designed for, like tracking specific subsets of a repo, automatically backfilling a database when adding new repos to monitor, or even mirroring the entire Atmosphere.
LiveStore
LiveStore is a next-generation state management framework based on reactive SQLite and git-inspired syncing (via event-sourcing). - LiveStore
Jazz - The database that syncs.
Jazz is a local-first relational database. It runs across your frontend, backend and our global storage cloud. Sync partial tables, durable streams and files, fast. Feels like simple reactive state.
proposal: ingest repo records #282
`sh.tangled.repo` records are one of the last few records that need atprotation (2-way sync between appview/pds). this one is tricky because we want consistency between knot state, appview state and PDS state. the path forward is described below: - migration to did/rkey syntax universally: in several places, we utilize `did/repo-name` as a globally unique identifier for a repository, we should migrate this to `did/rkey`: * for the tangled appview: this means reworking our ACLs, routers and DB to use `did/rkey` or ATURI as a globally unique identifier for a repo * for knots: this means changing paths on disk to be `did/rkey`, allowing git ops to `host:did/rkey` , updating ACLs, and XRPC endpoints * for spindles: this means updating ACLs and XRPC endpoints - once this is done, we can define ingestion logic for all services in the network to pull sh.tangled.repo records * for the tangled appview, this should create/delete/update a repo pointer record * for knots: this should create-or-ignore/delete-or-ignore/update-or-ignore a repo on disk. migration of repos needs to be thought out here. * for spindles: as above - define edge-case behaviors: * when referring to repos by rkey, it is possible for clever users to create duplicate records with the same repo-name, appview routers must handle this ambiguity with a new interstitial page that offers a redirect, knots must return an error upon push/pull on ambiguous git URLs * the `knot` cli can introduce a command `backfill` or `sync` to bring knot state up to sync with the rest of the world (repos, collaborators, pubkeys, ACLs etc.).
Conflict-Free Replicated Data Types
Replicating data under Eventual Consistency (EC) allows any replica to accept updates without remote synchronisation. This ensures performance and scalability in large-scale distributed systems (e.g., clouds). However, published EC approaches are ad-hoc and error-prone. Under a formal Strong Eventual Consistency (SEC) model, we study sufficient conditions for convergence. A data type that satisfies these conditions is called a Conflict-free Replicated Data Type (CRDT). Replicas of any CRDT are guaranteed to converge in a self-stabilising manner, despite any number of failures. This paper formalises two popular approaches (state- and operation-based) and their relevant sufficient conditions. We study a number of useful CRDTs, such as sets with clean semantics, supporting both add and remove operations, and consider in depth the more complex Graph data type. CRDT types can be composed to develop large-scale distributed applications, and have interesting theoretical properties.
Peer to Peer Databases
Peer to peer and decentralized applications have been popular lately, but almost all of them rely on synchronizing append only logs of operations (or an equivalent). This leads to performance degredation over time as data accumulates and needs to be processed on new peers before they can start querying and updating data within an existing swarm. This post will talk about how pre-indexed peer to peer databases can be built and how they can sidestep the need to process operation logs and query data more efficiently. The short of it is that with this new data structure, the network becomes your database instead of needing to build a database after loading everything from the network.
Cambria | Proceedings of the 8th Workshop on Principles and Practice of Consistency for Distributed Data
This summary was generated using automated tools and was not authored or reviewed by the article's author(s). It is provided to support discovery, help readers assess relevance, and assist readers from adjacent research areas in understanding the work. It is intended to complement the author-supplied abstract, which remains the primary summary of the paper. The full article remains the authoritative version of record. Click here to learn more.
Atproto for distributed systems engineers - AT Protocol
In this article we're going to explore atproto from the perspective of distributed backend engineering.
Games apps | at-store
Games apps grouped around a shared workflow, use case, or capability across the Bluesky ecosystem.
Personal sync servers for composable data
Exploring the possibilities of interoperability and composability for local-first software & data by using a personal sync server tied to an atproto identity
Personal sync servers for composable data
Exploring the possibilities of interoperability and composability for local-first software & data by using a personal sync server tied to an atproto identity
PostgreSQL Extensions or Protocols: Architecture Roulette
Should new infrastructure use PostgreSQL as a query engine, or aim for protocol compatibility instead?
