LoRA Without Regret

LoRA Without Regret

How LoRA matches full training performance more broadly than expected.

Library — cameron.stream

How LoRA matches full training performance more broadly than expected.

LoRA Learns Less and Forgets Less

Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.

Sebastian Raschka, PhD (@rasbt)

Is LoRA (Low Rank Adaptation) relevant in 2025 for reasoning models? I recently read "Tina: Tiny Reasoning Models via LoRA (https://arxiv.org/abs/2504.15777)", and it made me pause for a moment: when was the last time I heard someone excitedly talk/write about LoRA? LoRA was one of the most influential fine-tuning methods in the earlier LLM boom (as you may remember, I wrote about it a lot in recent years). The idea is simple but effective: avoid full model updates and instead inject a small number of trainable parameters for downstream tasks. This drastically reduces memory and compute costs. But in the age of ever-larger instruction-tuned models coupled with well-working distillation techniques (like popularized by DeepSeek-R1 etc), LoRA seemed to become more irrelevant recently. What about LoRA work for developing reasoning models? This paper tackles exactly that question. Instead of the usual supervised fine-tuning or instruction distillation pipeline, the authors use LoRA with reinforcement learning (RL) to improve reasoning capabilities. Specifically, they fine-tune a 1.5B base model using LoRA adapters while applying RL on reasoning benchmarks. Their baseline model is DeepSeek-R1-Distill-Qwen-1.5B, which is a model already fine-tuned for reasoning tasks. (I wish they started with the base Qwen-1.5B model; but this way, I guess they have more comparisons with other methods that further trained the DeepSeek-R1-Distill-Qwen-1.5B.) From there, the authors ran experiments across datasets, learning rates, LoRA ranks, and RL algorithms. Their best-performing model was trained on just 7k examples and cost just $9 to train. Even with hyperparameter sweeps and multiple ablations, the entire study cost just $526. So, how well does LoRA work? The top half of the results figure (highlighted in blue) compares models trained with LoRA-based RL versus standard RL (i.e., no LoRA). On every benchmark (AIME24, AIME25, AMC23, MATH500, GPAQ, Minerva), LoRA outperforms the regular RL baseline when applied to the same starting model. Insights from ablations 1) Surprisingly, the best-performing model came from the smallest dataset: just 7k examples from Open-RS. 2) The classic LoRA rank 16 emerged as the sweet spot, but ranks 8 and 32 also worked well. 3) It's nice that they included the recent Dr. GRPO (I recently discussed it in my latest Ahead of AI blog). It substantially reduces training time by length-normalizing rewards and addressing issues in GRPO Bottom line: Reasoning is certainly an interesting use case, and it's interesting (and a bit surprising) that LoRA does so well here. It might also be the first case where I've seen LoRA coupled with RL, which is another interesting aspect. LoRA certainly peaked in popularity 1-2 years ago, and more people now consider (more expensive) full-parameter updates (based on anecdotal perception); there's still a place for LoRA and LoRA-like methods. Let's not forget that one of the key advantages of LoRA is that it doesn't modify the underlying base model. This is key in applications where you either have lots of specialized use cases or lots of customers. For example, instead of storing 100 1B full-parameter tuned models, it would be much cheaper to store a 32B model with 100 sets of LoRA weights.

predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Sakana AI on Twitter / X

We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible.https://t.co/wGKDNhBcJXBy training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly… pic.twitter.com/gId3J6hgEr— Sakana AI (@SakanaAILabs) February 27, 2026

Radio Hobbyists, Rejoice! Good News for LoRa & Mesh

A set of radio devices and technologies are opening the doorway to new and revolutionary forms of communication. These have the potential to break down the over-reliance on traditional network

Training great LLMs entirely from ground up in the wilderness as a startup — Yi Tay

Chronicles of training strong LLMs from scratch in the wild

Why I don’t think AGI is right around the corner

Continual learning is a huge bottleneck

Ashe Build with Highest Winrate - LoL Runes, Items, and Skill Order

Ashe build with the highest winrate runes and items in every role. U.GG analyzes millions of LoL matches to give you the best LoL champion build. Patch 16.3

Ellora: Enhancing LLMs with LoRA - Standardized Recipes for Capability Enhancement

A Blog post by Asankhaya Sharma on Hugging Face

Sakana AI on Twitter / X

We’re excited to introduce Text-to-LoRA: a Hypernetwork that generates task-specific LLM adapters (LoRAs) based on a text description of the task. Catch our presentation at #ICML2025!Paper: https://t.co/2FRiVF1UXJCode: https://t.co/rx4G7dq1SWBiological systems are capable of… pic.twitter.com/UdUYfqRXBS— Sakana AI (@SakanaAILabs) June 12, 2025

Arcee AI | Use MergeKit to Extract LoRA Adapters from any Fine-Tuned Model

We show you how to use Arcee's MergeKit to extract LoRA adapters from fine-tuned models, then leverage the Hugging Face Hub to create a library of general and task-specific LoRA adapters.

Jiaxin Wen on Twitter / X

New post: "Generalization Dynamics of LM Pre-training"Most people (including me) assume that LMs smoothly mature from pattern-matching to generalizing. This mental model is wrong. The true dynamics are stranger, and far more fascinating! We call it Mode-Hopping. pic.twitter.com/SoNiCIKI2R— Jiaxin Wen (@jiaxinwen22) May 18, 2026

Gökdeniz Gülmez on Twitter / X

🚀 introducing mlx-embeddings-lora v1.0.5!„pip install -U mlx-embeddings-lora“Train your own text embedding models locally on Apple Silicon 🍏 using contrastive learning + LoRA fine-tuning — no GPUs, no cloud, just pure MLX speed.💡 Built on mlx-embeddings🧠 Supports LoRA…— Gökdeniz Gülmez (@ActuallyIsaak) November 13, 2025