The Short Answer
For 90% of domain adaptation use cases — teaching a model your terminology, document style, and knowledge base — QLoRA is the correct default. Dataset quality and hyperparameter tuning have 10x more impact than backbone bit-width.
The remaining 10% is where it gets interesting.
What QLoRA Actually Does
QLoRA (Quantized Low-Rank Adaptation) freezes the base model at 4-bit precision (NF4) and trains small LoRA adapter matrices in bf16. This means:
- Base model: frozen at 4-bit, never updated during training
- LoRA adapters: trained at full bf16 precision
- Memory cost: 70-80% less VRAM than full fine-tuning
- Training cost: single GPU (24GB VRAM) handles 14B parameter models
The key insight people miss: the adapters themselves train at full precision. The quantization only affects the frozen backbone weights. For domain adaptation — where the model needs to learn new surface patterns, terminology, and document formats — the adapter precision is what matters.
When QLoRA Is Sufficient
Domain Terminology and Style
Teaching a legal model to write like your firm’s partners. Teaching a medical model your clinical guideline format. Teaching a financial model your reporting templates.
These are surface pattern adaptations. The base model already understands language, reasoning, and world knowledge. Your adapter teaches it your specific vocabulary and formatting conventions. QLoRA handles this perfectly.
RAG-Grounded Question Answering
If your deployment uses retrieval-augmented generation (RAG), the retrieval pipeline does most of the heavy lifting. The model’s job is to synthesize retrieved passages — not to recall facts from parameters. QLoRA adapters easily learn to follow your RAG pipeline’s output format and attribution style.
Classification and Routing
Categorizing support tickets, routing requests, tagging documents. These tasks involve clear decision boundaries where 4-bit backbone quantization has negligible impact on accuracy.
When Full Precision Matters
DPO and RLHF Alignment
Direct Preference Optimization trains on subtle reward signals — “this response is slightly better than that response.” Quantization noise in the backbone can mask these preference gradients, making the model less responsive to alignment training.
If you’re doing DPO alignment to reduce refusals, improve tone, or steer behavior, consider full-precision training. The preference signal is too subtle for quantization artifacts.
Precision-Critical Numerical Output
Medical dosage calculations. Financial modeling. Engineering tolerances.
When your model produces numbers that humans act on, quantization rounding compounds across tokens. Each token generation samples from a distribution slightly distorted by 4-bit quantization. For conversational text, this distortion is invisible. For a sequence of numerical calculations, errors can compound.
Maxing Out a Small Model’s Capacity
If you’re pushing a 7B or 14B model to the ceiling of its capability — extracting every last percentage point of benchmark performance — then every bit of precision matters. The difference between QLoRA and full-precision on a 70B model is negligible. The difference on a 7B model handling complex reasoning can be measurable.
Safety-Critical Applications
When the model’s output directly affects patient care, legal outcomes, or physical safety, the conservative choice is full-precision training. Not because QLoRA is unreliable — but because the precision ceiling is higher, and for safety-critical applications, you want every margin available.
The Decision Framework
| Scenario | QLoRA Sufficient? | Notes |
|---|---|---|
| Domain terminology and style | Yes | Adapter learns surface patterns |
| RAG-grounded Q&A | Yes | Retrieval does the heavy lifting |
| Classification and routing | Yes | Clear decision boundaries |
| Complex multi-step reasoning | Mostly yes | Edge cases at reasoning boundaries |
| DPO / RLHF alignment | Sometimes no | Reward signal is subtle; quantization noise can mask preferences |
| Precision-critical numerical output | Maybe no | Rounding compounds across tokens |
| Maxing a small model’s capacity | No | Every bit of precision matters at the ceiling |
| Safety-critical deployment | Consider full | Higher precision ceiling for critical applications |
Training Infrastructure Comparison
| Method | VRAM Required (14B model) | Training Time | Hardware |
|---|---|---|---|
| QLoRA (NF4 + bf16 adapters) | 24 GB | 2-6 hours | Single A5000 / A6000 |
| Full LoRA (bf16) | 48-80 GB | 4-12 hours | A6000 / H100 |
| Full fine-tune (bf16) | 80-160 GB | 8-24 hours | Multi-H100 |
The cost difference is significant. QLoRA on a DGX Spark or single A6000 costs effectively nothing if you own the hardware. Full-precision training on rented H100s runs $2-4/GPU-hour.
Deployment: LoRA Adapters on VLLM
Both QLoRA and full-precision LoRA produce the same artifact: a LoRA adapter file (SafeTensors). VLLM loads these identically:
# VLLM with LoRA adapter (works for both QLoRA and full-precision adapters)
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.3-70B-Instruct \
--enable-lora \
--lora-modules my-adapter=/path/to/adapterOn NVIDIA hardware with VLLM, you get multi-LoRA hot-swap — multiple adapters loaded simultaneously, selected per-request. This means you can serve different domain adapters (legal, medical, financial) from a single base model.
Apple Silicon Considerations
If your deployment targets Apple Silicon (Mac Studio / Mac Pro):
- mlx-lm: supports LoRA adapters (one per server instance, not multi-LoRA)
- VLLM on Apple Silicon: full models only, no LoRA adapter support
- Ollama: requires merge + quantize (adapter baked into GGUF permanently)
For Apple Silicon deployments, you may need to merge the adapter into the base model and re-quantize. This is a one-time operation, but it means you lose the hot-swap capability.
Our Approach
At damore.ai, QLoRA via Unsloth is the default for our Custom LoRA Training package ($20K). Unsloth’s NF4 implementation is tighter than the original QLoRA paper, and we’ve validated it across healthcare, legal, financial, and education domains.
We offer full 16-bit training as a $3K-5K add-on for safety-critical, reasoning-intensive, or numerical-precision applications. This includes comparative benchmarking — we train both QLoRA and full-precision, run your evaluation suite on both, and you decide which to deploy based on the numbers.
Dataset quality is where we spend most of our effort. A well-curated dataset with QLoRA will outperform a mediocre dataset at full precision every time.