---
title: "What is LoRA? Low-rank adaptation for fine-tuning LLMs"
description: "LoRA fine-tunes an LLM by training two small low-rank matrices beside frozen weights. How it works, the parameter math, and why it adds no latency."
url: https://broadrouter.com/glossary/lora/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LoRA (low-rank adaptation)

Updated 2026-11-14

LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pre-trained model's weights and trains a pair of small low-rank matrices for selected layers. The learned update can be stored as a small adapter or merged into the weights, adding no inference latency.

Also called: low-rank adaptation, LoRA adapter, LoRA fine-tuning

LoRA made fine-tuning large models practical on ordinary hardware by training a tiny correction instead of the whole model.

## How it works

Hu and colleagues observed that the change fine-tuning makes to a weight matrix tends to have a low rank. So LoRA keeps the original matrix W₀ frozen and learns the change as the product of two thin matrices:

W = W₀ + BA, where B is d × r, A is r × k, and r is much smaller than d or k.

B starts at zero, so training begins from the unchanged model. The paper mostly applied LoRA to the attention query and value projections.

The parameter savings are easy to compute. For one 4,096 × 4,096 matrix:

|  | Parameters |
| --- | --- |
| Full update | 4,096 × 4,096 = 16,777,216 |
| LoRA, r = 8 | 8 × (4,096 + 4,096) = 65,536 |

That is about 0.4% of the full count. On GPT-3 175B the paper reports 10,000 times fewer trainable parameters, a 3× cut in GPU memory, and checkpoints shrinking from about 350 GB to about 35 MB.

## Why it matters

-   **Cheaper training.** Fewer trainable weights means less memory for optimiser state. Combined with [quantization](https://broadrouter.com/glossary/quantization/) of the frozen base, larger models fit on smaller GPUs.
-   **No extra latency.** Hugging Face’s PEFT guide notes adapters can be merged with `merge_and_unload()`, so serving costs the same as the base model.
-   **Many tasks, one base.** Because the base stays frozen, you can keep one copy of an [open-weight model](https://broadrouter.com/glossary/open-weight-model/) and swap small adapters per customer or task.
-   **Limits.** LoRA changes behaviour and style well. It is still [fine-tuning](https://broadrouter.com/glossary/fine-tuning/), so it needs good data and an eval.

## On BroadNet Inference

BroadNet’s [API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-lora&utm_content=body#models) does not document LoRA, adapter uploads or per-request adapter selection. You call fixed model IDs such as `broadnet/qwen3.8-27b`. Private endpoints and BroadNet’s Arabic fine-tunes are available on request, but the docs do not say how they were trained. If you plan to train adapters yourself, compare that against API prices with the post on [self-hosting vs an API](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/), and if you run both, a router such as [LiteLLM](https://broadrouter.com/guides/litellm-proxy-custom-provider/) can send traffic to either.

## Questions

### What does the rank r mean in LoRA?

It is the inner size of the two small matrices. A lower rank means fewer trainable parameters and a smaller adapter. The LoRA paper reports that ranks as low as 1 or 2 worked well on GPT-3 175B.

### Is LoRA as good as full fine-tuning?

The LoRA paper reports quality "on-par or better than fine-tuning" on RoBERTa, DeBERTa, GPT-2 and GPT-3, with far fewer trainable parameters. Test it on your own task.

### Can I upload a LoRA adapter to BroadNet Inference?

BroadNet's docs do not describe adapter uploads. Private endpoints and BroadNet's own fine-tunes are offered on request.

## See also

-   [Fine-tuning](https://broadrouter.com/glossary/fine-tuning/) (term)
-   [Quantization](https://broadrouter.com/glossary/quantization/) (term)
-   [Open-weight model](https://broadrouter.com/glossary/open-weight-model/) (term)
-   [Distillation (knowledge distillation)](https://broadrouter.com/glossary/distillation/) (term)
-   [Route LiteLLM to BroadNet Inference (SDK and proxy)](https://broadrouter.com/guides/litellm-proxy-custom-provider/) (guide)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)
-   [Open-weight vs closed AI models for business: cost, control, risk](https://broadrouter.com/blog/open-weight-vs-closed-models/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-lora&utm_content=page-cta) ·

**Sources**

-   [Hu et al. (2021), LoRA - Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)
-   [Hugging Face PEFT, LoRA conceptual guide](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora)
-   [BroadNet Inference API reference, model IDs](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-lora&utm_content=sources#models)
