Something I’ve been thinking about recently is how when I’ve worked on any kind of distributed system, including systems as simple as a web app with frontend and backend code, probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed. I came up with the following description of why I think this kind of programming requires so much effort: Everything is fragile narrow laggy asynchronous mismatched untrusted pipes. I think every programmer who’s worked on a networked system has encountered each of these issues, this is just my effort to coherently describe all of them in one place. I hope to prompt you to consider all the different hassles at once and think about how much harder/easier your job would be if you did/didn’t have to deal with these things. I think this is part of why web companies like Twitter seem to have so much lower impressiveness per engineer productivity than other places like game companies or SpaceX, although there’s other pieces to that puzzle. While part of the difficulty of distributed systems is inherent in physics, I think there’s lots of ideas for making each part of the problem easier, many already in common use, and I’ll try to mention lots of them. I hope that we as programmers continually develop more of these techniques and especially general implementations that simplify a problem. Like serialization libraries reducing the need for hand-written parsers/writers, I think there’s a lot of developer time out there to save by implementing generalized solutions where we currently painstakingly reimplement common patterns. I also think all these costs mean you should try really hard to avoid making your system distributed if you don’t have to.
Causal Islands
Realtime voice, video, and AI for developers - Daily
Daily is the team behind Pipecat. Ultra low latency, open source SDKs, and enterprise reliability since 2016.
Pipe Dreams: The life and times of Yahoo Pipes
In 2007, a small team at Yahoo! briefly changed how we program the internet. This is their story.
Peter van Hardenberg - Ink and Switch, Automerge
Brendan O'Brien (number 0) - Can Sync Be Network-optional?
Why Distributed Systems Are Hard to Develop — and How to Fix It
Here’s what’s missing when developing distributed systems today: Structure. Consistent, codified structure.
Cloud native Go: building reliable services in unreliable environments
What do Docker, Kubernetes, and Prometheus have in comm…
Causal Islands 2023
Home | cse223b
CSE 223B is a 4-unit graduate subject with lectures, paper discussions, a midterm, and technical report exercise, and a major term research project culminating in writing a research paper of your own. It will present abstractions and implementation techniques that facilitate the design of distributed systems–including both operating systems and sophisticated Internet servers–that can deal with the demands of real-world workloads. Topics will include efficient operating system primitives, high-performance network servers, load shedding, storage systems, security, and fault tolerance.
Datastar: Modern web dev, simplified
You love building web apps with Python, and HTMX got you excited about the hypermedia approach -- let the server drive the HTML, skip the JavaScript build step, keep things simple. But then you hit that last 10%: You need Alpine.js for interactivity, your state gets out of sync, and suddenly you're juggling two unrelated libraries that weren't designed to work together. What if there was a single 11-kilobyte framework that gave you everything HTMX and Alpine do, and more, with real-time updates, multiplayer collaboration out of the box, and performance so fast you're actually bottlenecked by the monitor's refresh rate? That's Datastar. On this episode, I sit down with its creator Delaney Gillilan, core maintainer Ben Croker, and Datastar convert Chris May to explore how this backend-driven, server-sent-events-first framework is changing the way full-stack developers think about the modern web.
Improving My Hardware Workflow With CI and Nix
One thing you might notice if you read my blog is that I do a lot of hardware projects. Generally, these include a board, some software, and possibly some extraneous components (like CAD design, such as a case). Software tooling is quite mature, but I find that hardware had traditionally lagged quite a bit behind. For a while, I just lived with this, but over the last few months I've put in a bunch of effort to improve my workflow.
Asynchronous Communication Is The Future Of Work | I Done This
Whether you fear its impersonal nature or thinks its the best thing since streaming television, asynchronous communication is here to stay Remote work is rising and online education is becoming more accepted and commonplace, both due to changing attitudes and the pandemic. These factors are only going to increase the…
Redundancy Protects Human Collaborations from Failure
In the name of efficiency, organizations and governments around the world are increasingly trimming redundancy---personnel or overlapping capabilities beyond what routine operations require. Drawing on distributed computing and cognitive science, we challenge this view. Using queueing simulations and a preregistered multiplayer experiment (N = 832 in 187 teams of 3–6), we show that redundancy improves performance through two distinct routes. Adding collaborators to a team improves resilience, with larger teams completing more tasks when members became temporarily unable to contribute. Giving team members overlapping capabilities also improved efficiency, with teams in which collaborators could perform multiple roles completing substantially more work. These findings suggest that efforts to improve short-term efficiency by reducing redundancy may come at a cost, quietly eroding a team’s capacity to withstand unexpected failures.
We're making Tailscale faster
Tailscale performance updates reduce memory use, increase throughput, and speed startup with multi-queue, writev, and netmap caching.
Adam Fish - Why Local First matters to businesses
It's Time To Rethink Everything by @t3dotgg