Track and document dplyr data pipelines. As you filter, mutate, and join your way through a data set, dtrackr seamlessly keeps track of your data flow and makes publication ready documentation of a data pipeline simple.
Kyle Mathews on Twitter / X
D2TS enables really powerful streaming joins, aggregations, etc. and will be powering a lot of Electric features in the future!This is an early preview but very cool tech. First Typescript implementation of Differential Dataflow afaik. https://t.co/i4qSJnVZWV— Kyle Mathews (@kylemathews) February 28, 2025
Snakemake
With Snakemake, data analysis workflows are defined via an easy to read, adaptable, yet powerful specification language on top of Python. Steps are defined by "rules", which denote how to generate a set of output files from a set of input files (e.g. using a shell command). Wildcards (in curly braces) provide generalization. Dependencies between rules are determined automatically.
Build Reproducible Analytical Pipelines with Nix
Streamlines the creation of reproducible analytical pipelines using default.nix expressions generated via the rix package for reproducibility. Define derivations in R, Python or Julia, chain them into a composition of pure functions and build the resulting pipeline using Nix as the underlying end-to-end build tool. Functions to plot the pipeline as a directed acyclic graph are included, as well as functions to load and inspect intermediary results for interactive analysis. User experience heavily inspired by the targets package.
elbersb/tidylog
Tidylog provides feedback about dplyr and tidyr operations. It provides wrapper functions for the most common functions, such as filter, mutate, select, and group_by, and provides detailed output for joins.
Lean Data Automation: A Principal Components Approach
Plus: A nimble, composable example with Github Actions, Hamilton, and Runhouse
Dynamic Data Science
<p>Use our Common Online Data Analysis Platform (CODAP) to explore dynamic data science activities and gain fluency in data moves to examine large datasets.</p>
Dremel: interactive analysis of web-scale datasets: Proceedings of the VLDB Endowment: Vol 3, No 1-2
Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system ...
Trusting AI with Your Data: Safe Automation from Branch to Production
DASL — Data-Addressed Structures & Links
A small set of simple, standard primitives to work on content-addressed data.
DASL — Data-Addressed Structures & Links
A small set of simple, standard primitives to work on content-addressed data.
Specification of graph translators with triple graph grammars
Data integration is a key issue for any integrated set of software tools. A typical CASE environment, for instance, offers tools for the manipulation of requirements and software design documents, and it provides more or less sophisticated assistance for keeping these documents in a consistent state. Up to now, almost all data consistency observing or preserving integration tools are hand-crafted due to the lack of generic implementation frameworks and the absence of adequate specification formalisms. Triple graph grammars are intended to fill this gap and to support the specification of interdependencies between graph-like data structures on a very high level. Furthermore, they are the fundamentals of a new machinery for the production of batch-oriented as well as incrementally working data integration tools.
protolab
A patchbay for your schemas. Bidirectional data transformations, drawn as circuits.
protolab
A patchbay for your schemas. Bidirectional data transformations, drawn as circuits.
