Fine-tune Llama 3.1 8B on your own data
Published 27 October 2026 · prices on this page are live
Llama 3.1 8B Instruct is a good first model to fine-tune: small enough for one 24 GB GPU with QLoRA, and strong enough to learn your format, your tone or your field. The same steps work for Qwen, Gemma and newer Llama models.
Fine-tune, or give the model your documents?
Fine-tuning changes how a model answers: the format, the style, the vocabulary of your field, a classification it should make. If the model must know facts that change, such as prices or your documents, retrieval (RAG) is usually the better tool; many projects use both.
Pick a method and a GPU
| Method | GPU memory for 8B | Good cards |
|---|---|---|
| QLoRA: the model in 4-bit, small adapters trained on top | about 10–16 GB, by sequence length and batch | 16–24 GB: RTX 4090, RTX 3090 |
| LoRA in 16-bit | about 20–30 GB | 40–48 GB: L40S, RTX A6000, A100 |
| Full fine-tuning | about 16 bytes per parameter with Adam, so 130 GB or more | Several 80 GB GPUs; not covered here |
QLoRA is where most people start: it costs the least and, for teaching a format or a style, comes close to full fine-tuning.
What a run costs
LoRA on about 10,000 examples for three epochs, estimated for a 7B model; an 8B model takes a little longer. On today's cheapest hosts:
| GPU | Typical time | USD | INR | NPR |
|---|---|---|---|---|
| A100 SXM4 | 2 h | USD 1.30 | INR 122 | NPR 192 |
| A100 PCIe | 2.2 h | USD 1.43 | INR 135 | NPR 212 |
| RTX 3090 | 5 h | USD 1.50 | INR 140 | NPR 220 |
| RTX 4090 | 3 h | USD 1.80 | INR 168 | NPR 264 |
| RTX 5090 | 2.5 h | USD 1.88 | INR 175 | NPR 278 |
1. Prepare your data
Write your examples as conversations, one per line in a JSONL file, each with a list of messages: an optional system message, the user's turn and the answer you want. A few hundred to a few thousand clean, consistent examples teach a format or a tone; quality matters far more than quantity. Hold back 5–10% to test on.
2. Rent the GPU
Choose the PyTorch + JupyterLab software and a 24 GB GPU, with about 60 GB of disk for the model, your data and checkpoints. Open JupyterLab from the console.
3. Get the model
Llama 3.1 is gated on Hugging Face: accept Meta's licence on the model page, create an access token, and log in on the machine with hf auth login (or huggingface-cli login on older versions). Then install the training libraries with pip install -U transformers peft trl bitsandbytes datasets.
4. Load it for QLoRA
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
base = "meta-llama/Llama-3.1-8B-Instruct"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(base)
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM")
Then train with TRL's SFTTrainer on your JSONL file: it applies Llama's chat template to the messages. Two or three epochs at a learning rate around 2e-4 is a common start for LoRA, with a batch that fits the card and gradient accumulation for the rest. Watch the loss on your held-back examples: when it stops falling, stop.
5. Test it and run it
- Compare the base model and yours on the held-back prompts, side by side, before you trust it.
- Save the LoRA adapter: it is small, tens to a few hundred megabytes, so download it or push it to a private Hugging Face repository.
- Serve it with vLLM, which loads the base model and your adapter behind an OpenAI-compatible API:
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-lora --lora-modules mine=./adapter. The vLLM software on PowerPod has it ready. - For Ollama, merge the adapter into the model, convert it to GGUF with llama.cpp, and import it with a Modelfile.
Llama 3.1 comes with Meta's Llama 3.1 Community License: read it before you use the model, or your fine-tune, in a product.
Questions
Can I fine-tune Llama 3.1 8B on a 16 GB GPU?
Yes, with QLoRA, sequences up to about 2,048 tokens and a small batch with gradient accumulation. A 24 GB card gives more room for longer examples.
How many examples do I need?
To teach a format or a tone, a few hundred to a few thousand good examples often do. Consistent, correct examples matter more than a large, noisy set.
How long does it take?
About two to five hours on a 24 GB card for 10,000 examples and three epochs of LoRA, and about an hour on an H100.
Can I use Qwen or Gemma instead?
Yes. The same steps work; pick the base model that does best on your own test prompts before you train.