A series of case studies using simulated datasets to explore successful recipes for transfer learning
Selecting synthetic data for successful simulation-based transfer learning in dynamical biological systems
Accurate prediction of the temporal dynamics of biological systems is crucial for informing timely and effective interventions, e.g., in ecological or epidemiological contexts, or for treatment adjustments in therapy. While machine learning has proven its capabilities in generalizing the underlying non-linear dynamics of such systems, unlocking its predictive power is often restrained by the limited availability of large, curated datasets. To supplement real-world data, informing machine learning by transfer learning with synthetic data derived from simulations using ordinary differential equations has emerged as a promising solution. However, the success of this approach highly depends on the designed characteristics of the synthetic data.
Survey: How will neuroscience training need to change?
Training in computational neuroscience, data science and statistics is needed but must be balanced with grounding in the scientific method, say respondents.
Active Learning with Statistical Models
Understanding Deep Learning Algorithms that Leverage Unlabeled Data, Part 2: Contrastive Learning
Theoretical analysis of contrastive learning algorithms for leveraging unlabeled data.
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
Current deep neural networks(DNNs) can easily overfit to biased training data with corrupted labels or class imbalance. Sample re-weighting strategy is commonly used to alleviate this issue by designing a weighting function mapping from training loss to sample weight, and then iterating between weight recalculating and classifier updating. Current approaches, however, need manually pre-specify the weighting function as well as its additional hyper-parameters. It makes them fairly hard to be generally applied in practice due to the significant variation of proper weighting schemes relying on the investigated problem and training data. To address this issue, we propose a method capable of adaptively learning an explicit weighting function directly from data. The weighting function is an MLP with one hidden layer, constituting a universal approximator to almost any continuous functions, making the method able to fit a wide range of weighting function forms including those assumed in conventional research. Guided by a small amount of unbiased meta-data, the parameters of the weighting function can be finely updated simultaneously with the learning process of the classifiers. Synthetic and real experiments substantiate the capability of our method for achieving proper weighting functions in class imbalance and noisy label cases, fully complying with the common settings in traditional methods, and more complicated scenarios beyond conventional cases. This naturally leads to its better accuracy than other state-of-the-art methods.
Quantifying the Carbon Emissions of Machine Learning
From an environmental standpoint, there are a few crucial aspects of training a neural network that have a major impact on the quantity of carbon that it emits. These factors include: the location of the server used for training and the energy grid that it uses, the length of the training procedure, and even the make and model of hardware on which the training takes place. In order to approximate these emissions, we present our Machine Learning Emissions Calculator, a tool for our community to better understand the environmental impact of training ML models. We accompany this tool with an explanation of the factors cited above, as well as concrete actions that individual practitioners and organizations can take to mitigate their carbon emissions.
Designing machine learning systems: an iterative process for production-ready applications
"Machine learning systems are both complex and unique. Complex because they consist of many different components and involve many different stakeholders. Unique because they're data dependent, with data varying wildly from one use case to the next. In this book, you'll learn a holistic approach to designing ML systems that are reliable, scalable, maintainable, and adaptive to changing environments and business requirements. Author Chip Huyen, co-founder of Claypot AI, considers each design decision--such as how to process and create training data, which features to use, how often to retrain models, and what to monitor--in the context of how it can help your system as a whole achieve its objectives. The iterative framework in this book uses actual case studies backed by ample references."--Amazon.com
Better Training Data Attribution via Better Inverse Hessian-Vector Products
Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance.
Dataset Distillation
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called {\em dataset distillation}: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to {\em synthesize} a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress $60,000$ MNIST training images into just $10$ synthetic {\em distilled images} (one per class) and achieve close to original performance with only a few steps of gradient descent, given a particular fixed network initialization. We evaluate our method in a wide range of initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods in most settings.
NeuroAI
Neuroscience, cognitive science, and AI are all questing for principles that help generalization. Learn more through a live, synchronous program designed for focused, hands-on learning.
Deconstructing the Machine Learning Value Chain
This essay offers a materialist analysis of machine learning in the form of a critical intervention that deconstructs the value chain of this technological para
Introducing dtoolAI — dtoolAI 0.1.0 documentation
dtoolAI is a Python library to make reproducible AI model training and use easier. The dtoolAI package provides:
Pace Layering: How Complex Systems Learn and Keep Learning · Journal of Design and Science
Pace layers provide many-leveled corrective, stabilizing feedback throughout the system. It is in the contradictions between these layers that civilization finds its surest health. I propose six significant levels of pace and size in a robust and adaptable civilization.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
AI’s Memorization Crisis
Large language models don’t “learn”—they copy. And that could change everything for the tech industry.
RSL for RAG/search makes sense, RSL for training doesn't · rslstandard · Discussion #1
The top of https://rslcollective.org/license says: Below are instructions for making your content available for training AI models through the collective licensing terms negotiated by the RSL Colle...