This is a repo with links to everything you'd ever want to learn about data engineering
Introduction to the Field of Data Engineering - 📖 Patterns of Data Engineering
Explore the evolution and challenges of data engineering from its roots in business intelligence to its current status as a critical field in technology. This chapter offers a comprehensive introduction, covering the historical development, current trends, and future directions of data engineering, alongside personal insights from the author's journey in the field.
Show me the data - Design Notes
Data Integration
This book is an introduction to the problem of data integration and a rigorous account of one of the leading approaches to solving this problem.
Data Science in 2020: Computing, Curricula, and Challenges for the Next 10 Years
In the past 10 years, new data science courses and programs have proliferated at the collegiate level. As faculty and administrators enter the race to provide data science training and attract new ...
Data Resources
OSoMe is home to a number of datasets that are available to the IU research community. Many of these data resources, including the Botometer Pro API and the...
Datasette
Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API.
What is data oriented design?
I was reading this article, and this guy goes on talking about how everyone can greatly benefit from mixing in data oriented design with OOP. He doesn't show any code samples, however. I googled t...
Foundations of Data Science for Students in Grades K-12
On September 13 and 14, 2022, the Board on Science Education at the National Academies of Sciences, Engineering, and Medicine held a workshop entitled Foundations of Data Science for Students in Grades K–12. Speakers and participants explored the rapidly growing field of K-12 data science education, by surveying the current landscape, surfacing what is known, and identifying what is needed to support student learning, develop curriculum and tools, and prepare educators. To support these conversations, four papers were commissioned and discussed during the workshop. This publication summarizes the presentations and discussion of the workshop.
Designing Data-Intensive Applications
Data is at the center of many challenges in system design today. Difficult issues need to be figured out, such as scalability, consistency, reliability, efficiency, and... - Selection from Designing Data-Intensive Applications [Book]
New data agents across the Agentic Data Cloud | Google Cloud Blog
Learn about new data agents and tools for business analysts, data scientists, and database admins to integrate with the Agentic Data Cloud.
Kyle Mathews on Twitter / X
D2TS enables really powerful streaming joins, aggregations, etc. and will be powering a lot of Electric features in the future!This is an early preview but very cool tech. First Typescript implementation of Differential Dataflow afaik. https://t.co/i4qSJnVZWV— Kyle Mathews (@kylemathews) February 28, 2025
Algorithmic Data Minimization for Machine Learning over...
Machine learning can analyze vast amounts of data generated by IoT devices to identify patterns, make predictions, and enable real-time decision-making. By processing sensor data, machine learning...
Drawing a map of distributed data systems — Martin Kleppmann’s blog
How we created an illustrated guide to help you find your way through the data landscape.
Tutorial on Sheaves in Data Analytics : Lecture 1
Designing Data-Intensive Applications
Data is at the center of many challenges in system desi…
here it is folks! lots of details to still nail down, but this is roughly where our heads are at for the design of a permissioned data protocol give it a read and let me know your thoughts!
Permissioned Data Diary 4: The Big Picture
dholms.leaflet.pub