You can fine-tune a 7B or 8B parameter LLaMA model on a single consumer GPU in under 2 hours. No cloud cluster, no $10K compute bill, no ML PhD. QLoRA (Quantized Low-Rank Adaptation) makes this possible by freezing the base model weights at 4-bit precision and training only a small set of adapter weights — typically 0.1–1% of the total parameters. The result: a model that knows your domain, speaks your brand voice, and handles your specific tasks better than any prompt-engineered base model. Here's how to do it, start to finish.
Fine-tune when: your task requires a specific output format that prompt engineering can't reliably produce, your domain has specialized vocabulary the base model doesn't handle well, you need consistent brand voice across thousands of outputs, or you want to reduce inference costs by training a small model to do what previously required a large one.
Don't fine-tune when: you can solve the problem with a good system prompt (try that first — it's faster), your dataset is under 100 examples (you'll overfit), your task changes frequently (fine-tuned models are static — you'd need to retrain), or you need the model to cite real-time information (fine-tuning teaches patterns, not facts).
The decision tree: Can you solve it with a prompt? → Do that. Can you solve it with few-shot examples in the prompt? → Do that. Does it need consistent behavior across 10,000+ inputs at low cost? → Fine-tune.
Fine-tuning data is a JSONL file where each line contains an input-output pair in the model's chat format. For LLaMA, this means structured conversations with system, user, and assistant roles. Minimum viable dataset: 200–500 high-quality examples. Sweet spot: 1,000–5,000. Beyond 10,000, you get diminishing returns for most tasks.
Quality matters more than quantity. 500 expertly crafted examples beat 5,000 mediocre ones. Each example should represent the exact behavior you want: if you're training a customer support bot, your examples should be real support conversations (anonymized) with the responses written the way you want the model to respond — not the way it currently responds.
Common data sources: your existing chat logs (filtered for quality), manually written examples of ideal responses, outputs from a stronger model (Claude Opus) that you've edited and approved (distillation), or synthetic data generated from templates with human review. Avoid scraping random internet text — that's what pretraining already did. Your fine-tune data needs to be domain-specific and high-signal.
You need: Python 3.10+, PyTorch 2.x with CUDA support, the transformers, peft, bitsandbytes, trl, and datasets libraries from Hugging Face. Install everything with pip install torch transformers peft bitsandbytes trl datasets accelerate. Hardware: an NVIDIA GPU with 16GB+ VRAM (RTX 4080, 4090, 5080, or 5090). An 8B model with QLoRA fits comfortably in 12GB; a 13B model needs 16–20GB.
If you're on a Mac with Apple Silicon: use MLX instead of PyTorch. The mlx-lm library supports LoRA fine-tuning natively on M-series chips. Performance is 60–70% of an equivalent NVIDIA GPU, but it works without the CUDA dependency. For M4 Pro with 24GB unified memory, you can fine-tune models up to 13B parameters.
The key QLoRA parameters: r (rank) — typically 16, 32, or 64. Higher rank = more trainable parameters = more capacity to learn new behaviors, but more VRAM and slower training. Start with r=32. lora_alpha — scaling factor, usually set to 2x the rank (so 64 if r=32). target_modules — which layers to adapt. For LLaMA: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. Training all attention and MLP projections gives the best results. lora_dropout — set to 0.05–0.1 to prevent overfitting on small datasets.
Training hyperparameters: learning rate 2e-4 (the most forgiving default), batch size 4–8 with gradient accumulation to simulate larger batches, 3 epochs (2 if your dataset is large, 5 if it's tiny), and cosine learning rate schedule with 10% warmup. These defaults work for 90% of use cases. Don't over-tune hyperparameters until you've confirmed the approach works at all.
With the Hugging Face SFTTrainer from the trl library, the training script is about 50 lines of Python. Load the base model in 4-bit (NF4 quantization), attach the LoRA adapter, point at your dataset, set the training arguments, and call trainer.train(). On an RTX 4090, a 1,000-example dataset against an 8B model completes in 30–45 minutes. A 5,000-example dataset takes 2–3 hours.
Monitor the training loss. It should decrease steadily through epoch 1, flatten in epoch 2, and stabilize in epoch 3. If it's still decreasing sharply at the end of epoch 3, you might benefit from more epochs. If it hits near-zero by mid-epoch 2, you're overfitting — reduce epochs or add more diverse training data.
Test your fine-tuned model on a held-out set of 50–100 examples that weren't in the training data. Compare outputs against the base model on the same inputs. If the fine-tuned model consistently matches your desired output format and quality, you're done. If specific failure modes persist, add 50–100 examples targeting those failures and retrain.
Once satisfied, merge the LoRA adapter into the base model weights. This produces a single model file that runs without the adapter overhead — slightly faster inference and simpler deployment. Export to GGUF format for use with Ollama or llama.cpp for production deployment.
QADIR OS ships with 8 custom LoRA adapters — purpose-trained for personality, tool use, memory handling, and humility (yes, we fine-tuned a model to be less sycophantic). The agentic loop selects the right adapter at runtime based on the task type. The same fine-tuning pipeline is available to QADIR OS users: feed it your data, it trains on your local GPU, and your agent learns your domain. No cloud training required.
Train a model that knows your business — on your own GPU. QADIR OS includes a built-in fine-tuning pipeline for domain-specific AI. Try 164+ free AI tools, or join early access — no card required.