LoRA Without Regret

How LoRA matches full training performance more broadly than expected.

LoRA Without Regret

How LoRA matches full training performance more broadly than expected.

Sakana AI on Twitter / X

We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible.https://t.co/wGKDNhBcJXBy training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly… pic.twitter.com/gId3J6hgEr— Sakana AI (@SakanaAILabs) February 27, 2026

LoRA Learns Less and Forgets Less

Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.

Sakana AI on Twitter / X

We’re excited to introduce Text-to-LoRA: a Hypernetwork that generates task-specific LLM adapters (LoRAs) based on a text description of the task. Catch our presentation at #ICML2025!Paper: https://t.co/2FRiVF1UXJCode: https://t.co/rx4G7dq1SWBiological systems are capable of… pic.twitter.com/UdUYfqRXBS— Sakana AI (@SakanaAILabs) June 12, 2025

Post-Training 50x Faster

We're announcing Trellis, the fastest open-source post-training code for Kimi K2 Thinking

Sebastian Raschka, PhD (@rasbt)

Is LoRA (Low Rank Adaptation) relevant in 2025 for reasoning models? I recently read "Tina: Tiny Reasoning Models via LoRA (https://arxiv.org/abs/2504.15777)", and it made me pause for a moment: when was the last time I heard someone excitedly talk/write about LoRA? LoRA was one of the most influential fine-tuning methods in the earlier LLM boom (as you may remember, I wrote about it a lot in recent years). The idea is simple but effective: avoid full model updates and instead inject a small number of trainable parameters for downstream tasks. This drastically reduces memory and compute costs. But in the age of ever-larger instruction-tuned models coupled with well-working distillation techniques (like popularized by DeepSeek-R1 etc), LoRA seemed to become more irrelevant recently. What about LoRA work for developing reasoning models? This paper tackles exactly that question. Instead of the usual supervised fine-tuning or instruction distillation pipeline, the authors use LoRA with reinforcement learning (RL) to improve reasoning capabilities. Specifically, they fine-tune a 1.5B base model using LoRA adapters while applying RL on reasoning benchmarks. Their baseline model is DeepSeek-R1-Distill-Qwen-1.5B, which is a model already fine-tuned for reasoning tasks. (I wish they started with the base Qwen-1.5B model; but this way, I guess they have more comparisons with other methods that further trained the DeepSeek-R1-Distill-Qwen-1.5B.) From there, the authors ran experiments across datasets, learning rates, LoRA ranks, and RL algorithms. Their best-performing model was trained on just 7k examples and cost just $9 to train. Even with hyperparameter sweeps and multiple ablations, the entire study cost just $526. So, how well does LoRA work? The top half of the results figure (highlighted in blue) compares models trained with LoRA-based RL versus standard RL (i.e., no LoRA). On every benchmark (AIME24, AIME25, AMC23, MATH500, GPAQ, Minerva), LoRA outperforms the regular RL baseline when applied to the same starting model. Insights from ablations 1) Surprisingly, the best-performing model came from the smallest dataset: just 7k examples from Open-RS. 2) The classic LoRA rank 16 emerged as the sweet spot, but ranks 8 and 32 also worked well. 3) It's nice that they included the recent Dr. GRPO (I recently discussed it in my latest Ahead of AI blog). It substantially reduces training time by length-normalizing rewards and addressing issues in GRPO Bottom line: Reasoning is certainly an interesting use case, and it's interesting (and a bit surprising) that LoRA does so well here. It might also be the first case where I've seen LoRA coupled with RL, which is another interesting aspect. LoRA certainly peaked in popularity 1-2 years ago, and more people now consider (more expensive) full-parameter updates (based on anecdotal perception); there's still a place for LoRA and LoRA-like methods. Let's not forget that one of the key advantages of LoRA is that it doesn't modify the underlying base model. This is key in applications where you either have lots of specialized use cases or lots of customers. For example, instead of storing 100 1B full-parameter tuned models, it would be much cheaper to store a 32B model with 100 sets of LoRA weights.

Brandon Stewart on Twitter / X

1/ New @Nature! We study how powerful institutions shape the information environment for LLMs. Commercial LLM training is opaque, so we trace a path from state-coordinated media -> training data -> model responses. pic.twitter.com/5LdFvzbFaf— Brandon Stewart (@b_m_stewart) May 13, 2026

Login | TrainingPeaks

Log in to your TrainingPeaks athlete edition or coach edition to track, analyze and plan.

llm-datasets

Curated list of datasets and tools for post-training.

predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Gökdeniz Gülmez on Twitter / X

🚀 introducing mlx-embeddings-lora v1.0.5!„pip install -U mlx-embeddings-lora“Train your own text embedding models locally on Apple Silicon 🍏 using contrastive learning + LoRA fine-tuning — no GPUs, no cloud, just pure MLX speed.💡 Built on mlx-embeddings🧠 Supports LoRA…— Gökdeniz Gülmez (@ActuallyIsaak) November 13, 2025

Training great LLMs entirely from ground up in the wilderness as a startup — Yi Tay

Chronicles of training strong LLMs from scratch in the wild

Extracting Training Data from Large Language Models

Nicholas Carlini, Google; Florian Tramèr, Stanford University; Eric Wallace, UC Berkeley; Matthew Jagielski, Northeastern University; Ariel Herbert-Voss, OpenAI and Harvard University; Katherine Lee and Adam Roberts, Google; Tom Brown, OpenAI; Dawn Song, UC Berkeley; Úlfar Erlingsson, Apple; Alina Oprea, Northeastern University; Colin Raffel, Google