BroadRouter
Get started

LoRA (low-rank adaptation)

Updated

LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pre-trained model's weights and trains a pair of small low-rank matrices for selected layers. The learned update can be stored as a small adapter or merged into the weights, adding no inference latency.

Also called: low-rank adaptation, LoRA adapter, LoRA fine-tuning

LoRA made fine-tuning large models practical on ordinary hardware by training a tiny correction instead of the whole model.

How it works

Hu and colleagues observed that the change fine-tuning makes to a weight matrix tends to have a low rank. So LoRA keeps the original matrix W₀ frozen and learns the change as the product of two thin matrices:

W = W₀ + BA, where B is d × r, A is r × k, and r is much smaller than d or k.

B starts at zero, so training begins from the unchanged model. The paper mostly applied LoRA to the attention query and value projections.

The parameter savings are easy to compute. For one 4,096 × 4,096 matrix:

Parameters
Full update 4,096 × 4,096 = 16,777,216
LoRA, r = 8 8 × (4,096 + 4,096) = 65,536

That is about 0.4% of the full count. On GPT-3 175B the paper reports 10,000 times fewer trainable parameters, a 3× cut in GPU memory, and checkpoints shrinking from about 350 GB to about 35 MB.

Why it matters

  • Cheaper training. Fewer trainable weights means less memory for optimiser state. Combined with quantization of the frozen base, larger models fit on smaller GPUs.
  • No extra latency. Hugging Face’s PEFT guide notes adapters can be merged with merge_and_unload(), so serving costs the same as the base model.
  • Many tasks, one base. Because the base stays frozen, you can keep one copy of an open-weight model and swap small adapters per customer or task.
  • Limits. LoRA changes behaviour and style well. It is still fine-tuning, so it needs good data and an eval.

On BroadNet Inference

BroadNet’s API reference does not document LoRA, adapter uploads or per-request adapter selection. You call fixed model IDs such as broadnet/qwen3.8-27b. Private endpoints and BroadNet’s Arabic fine-tunes are available on request, but the docs do not say how they were trained. If you plan to train adapters yourself, compare that against API prices with the post on self-hosting vs an API, and if you run both, a router such as LiteLLM can send traffic to either.

Questions

What does the rank r mean in LoRA?

It is the inner size of the two small matrices. A lower rank means fewer trainable parameters and a smaller adapter. The LoRA paper reports that ranks as low as 1 or 2 worked well on GPT-3 175B.

Is LoRA as good as full fine-tuning?

The LoRA paper reports quality "on-par or better than fine-tuning" on RoBERTa, DeBERTa, GPT-2 and GPT-3, with far fewer trainable parameters. Test it on your own task.

Can I upload a LoRA adapter to BroadNet Inference?

BroadNet's docs do not describe adapter uploads. Private endpoints and BroadNet's own fine-tunes are offered on request.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity