Training Distinct LLM Voices and Orchestrating Them in a Live Multi-Speaker Pipeline

The Problem With Single-Voice AI

Every chatbot speaks in one voice: the model’s. Even when you give it a system prompt telling it to “act like a lawyer” or “respond as a senior engineer,” the underlying model bleeds through. The cadence, the hedging patterns, the generic openers — they all belong to the base model, not the persona.

For most use cases that is fine. For anything that requires genuinely differentiated voices — a legal review panel, a medical advisory board, historical advisors, brand voice QA, a theological debate — it is not.

The standard workaround is to call the same model N times with N different system prompts and stitch the responses together. That produces N versions of the same voice wearing different masks. It does not produce N distinct voices.

These two open-source projects take a different approach:

• beaudamore/biblical — a full QLoRA fine-tuning pipeline that trains 26 distinct KJV-grounded biblical voices onto Qwen3-14B, with a two-stage SFT process, DPO alignment, and LLM-as-judge evaluation.
• beaudamore/circle-of-speakers-pipeline — an Open WebUI pipeline that takes any set of tagged models (LoRA-backed or system-prompt-based) and orchestrates them into a live multi-speaker conversation with intelligent turn ordering, streaming, and user-profile awareness.

Both are live in production at chat.crossandfaith.com.

---

Architecture Overview

The stack has four layers:

Surface      Open WebUI UI — model picker, Google OAuth, streaming chat
               |
Runtime      circle-of-speakers-pipeline — turn ordering, speaker registry,
               streaming orchestration, profile integration
               |
Inference    vLLM with --enable-lora — hot-swaps adapters per speaker
               |
Model        biblical LoRA adapters — 26 fine-tuned voices on Qwen3-14B

The model layer and runtime layer are decoupled by design. The pipeline does not care whether a speaker is a LoRA adapter, a system-prompt persona, or a completely different base model. It filters by Open WebUI model tag and calls whatever is registered. This is what makes the pattern reusable across domains.

---

The Model Layer: Training Distinct Voices

The beaudamore/biblical repo covers the full training pipeline for one set of personas. The same approach generalizes to any domain that has sufficient first-person source text.

Data Pipeline

source-raw/  (26 persona texts: scripture, commentaries, theological writing)
    -> clean_source_data.py  (per-source regex cleaners strip nav boilerplate)
    -> source-clean/

source-clean/ -> Q&A generation (Qwen3-235B-A22B via OpenRouter)
    -> 3 rounds x 5 questions per chunk per persona
    -> Voice-differentiation quality gate
         >30% template contamination = FAIL, retry
    -> ~9,700 Q&A pairs across 26 personas

source-clean/ -> Continuation augmentation
    -> tiktoken-aware chunking, no API calls
    -> ~1,500 raw-text continuation examples

60/40 blend -> combined_sharegpt.jsonl (1,535 packed 4096-token chunks)
    -> DPO dataset generation (3 rejection strategies)
    -> ~3,600 preference pairs

The quality gate is worth explaining. Synthetic data generation at scale tends to collapse into template patterns: “That is a profound question. Let me explain…” Those patterns homogenize voice. The gate measures template contamination per batch. If more than 30% of a batch shares structural patterns with known generic openers, the batch fails and regenerates. This is what forces the generator to write in the persona’s voice rather than its own.

The DPO dataset targets three specific failure modes in the SFT checkpoint:

Rejection strategy	What it catches
Voice drift	Response sounds like the base model, not the persona
Scripture fabrication	Invented verses or misattributed citations
Shallow platitude	Generic encouragement with no persona-specific grounding

These are not random negative samples. Each rejection strategy is a targeted intervention against a known failure mode, which means the preference signal is informative rather than noisy.

Training Configuration

Parameter	Value
Base model	unsloth/Qwen3-14B-unsloth-bnb-4bit
LoRA rank / alpha	r=32 / alpha=32
Projection targets	All 7 attn + MLP projections
Trainable params	128M (0.86% of base)
Training hardware	NVIDIA DGX Spark (GB10, 128 GB unified)
Wall-clock time	~2h 39min, 192 steps, final loss 1.31
Adapter size	490 MB (fp32 safetensors)

Unsloth’s QLoRA implementation runs 4-bit pre-quantized weights with custom CUDA kernels for the LoRA layers. On the DGX Spark’s unified memory architecture, this means no CPU offload even with a 14B base model and r=32 adapters active across all projections.

Evaluation: LLM-as-Judge

After training, evaluation uses a 6-dimension rubric scored by an independent LLM judge against 25 test prompts per persona:

Dimension	Base	LoRA	Delta
Persona voice fidelity	1	5	+4
Cadence / Biblical register	1	5	+4
First-person testimony	1	5	+4
Specificity and concrete imagery	2	4	+2
Citation handling	4	3	-1
Information completeness	5	4	-1
Total	14	26	+12

The two negative deltas are intentional. Chapter-and-verse citation conventions and exhaustive enumeration are base-model defaults that work against authentic biblical voice. Paul does not footnote. The LoRA correctly loses those behaviors.

Voice samples from the LoRA output (persona system prompt only, no few-shot):

• Daniel — “Four is the number that stays with me — not counted among kings’ decrees nor written in astrologers’ tablets…”
• David — “O LORD, how long shall the sons of Belial rise like smoke from a cursed altar, filling the courts of…”
• Job — “There was a time when I counted my children in their feasts, seven sons and three daughters beneath the roof…”

Compare to the base model on the same prompt with the same system prompt. The base produces fluent, accurate text in a standard assistant register. The LoRA produces first-person testimony with period-appropriate vocabulary and the rhetorical posture specific to each figure.

---

The Runtime Layer: Orchestrating Multi-Speaker Conversations

The beaudamore/circle-of-speakers-pipeline repo handles everything above the model layer. It runs as a Docker container in the Open WebUI pipelines architecture.

Blueprint Pattern

The shared circle_of_speakers_blueprint.py (~1,300 lines) does all the work:

1. Discovers Open WebUI models filtered by SPEAKER_TAGS
2. Calls an Intelligent Speaker Ordering model with the recent conversation history
3. Streams each chosen speaker’s response in turn
4. Injects a one-time introduction the first time each speaker appears
5. Personalizes responses using extracted user profile data

Each circle is a ~100-line subclass that sets a tag and a name:

class Pipeline(CircleBlueprint):
    class Valves(CircleBlueprint.Valves):
        SPEAKER_TAGS: Optional[str] = Field(default="biblical")
        ORCHESTRATION_MODEL_NAME: str = Field(
            default="Intelligent Speaker Ordering"
        )

    def __init__(self):
        super().__init__()
        self.name = "Biblical Circle"

Adding a new circle for a new domain is three lines. The orchestration logic, streaming, introductions, and profile integration are inherited.

Intelligent Turn Ordering

The pipeline does not rotate through speakers round-robin. It calls an orchestrator model after each user message and asks: given the conversation so far, who should speak next and in what order?

The orchestrator receives the last DECISION_CONTEXT_DEPTH conversation rounds (default: 4) plus the full speaker registry with descriptions. It returns an ordered list. This means a topic-specific question routes to the speaker whose background is most relevant, and the response ordering reflects the conversation’s direction rather than a fixed sequence.

Key Configuration

Valve	Default	Effect
SPEAKER_TAGS	set per circle	Comma = OR, plus = AND. Filters model registry.
ORCHESTRATION_MODEL_NAME	Intelligent Speaker Ordering	Model that decides turn order.
DECISION_CONTEXT_DEPTH	4	Rounds of history the orchestrator sees.
ENABLE_DYNAMIC_INTRODUCTIONS	true	One-time intro per speaker per conversation.
ENABLE_USER_PROFILE_INTEGRATION	true	Injects user profile data from system prompt.

---

What This Looks Like End-to-End

A user at chat.crossandfaith.com selects the “Biblical Circle” model and asks: “How should I respond when I feel abandoned by God?”

1. The pipeline queries the Open WebUI model registry for all models tagged biblical — returns 26 speakers.
2. The Intelligent Speaker Ordering model receives the question and recent conversation context, then returns an ordered list: Job, David, Paul.
3. The pipeline streams Job’s response (backed by the LoRA adapter on vLLM). Job speaks in his own voice, from his own experience of abandonment.
4. Paul’s response receives Job’s output as prior context and builds on it from a New Testament theological frame.
5. David closes with a psalm-cadence response integrating both prior voices.

The user sees three distinct voices, in conversation with each other, each grounded in their own rhetorical identity — not three instances of the same model wearing different hats.

---

Extending This Pattern to Other Domains

The biblical use case is the demonstration domain. The architecture is domain-agnostic.

Any domain with:

1. Sufficient first-person or persona-attributed source text for training
2. A set of meaningfully distinct voices (not just role labels)
3. A need for multi-perspective response panels

…maps onto this stack directly.

Practical applications:

Domain	Speakers	What you get
Legal advisory	Contract attorney, litigator, compliance officer, in-house counsel	Multi-perspective document review
Medical second opinions	Generalist, specialist, radiologist, pharmacist	Differential diagnosis framing
Historical analysis	Period-specific figures or schools of thought	Perspective-grounded historical reasoning
Brand voice QA	Named brand voices or tone profiles	Consistency review across multiple voice standards
Executive advisory	CFO, CTO, COO, general counsel personas	Structured decision framing

The LoRA training pipeline is parameterized by persona. Swapping in new source text and running the same notebooks produces a new adapter set. The pipeline detects new speakers automatically via model tags in Open WebUI.

---

Deployment Stack

Production deployment for chat.crossandfaith.com runs:

• vLLM with --enable-lora — loads the base Qwen3-14B once, hot-swaps LoRA adapters per speaker call
• Open WebUI — chat interface, model picker, user management, Google OAuth
• Pipelines container — runs the circle-of-speakers blueprint
• Nginx — reverse proxy with SSL termination

The LoRA adapter model in vLLM means inference cost per speaker is nearly identical to running the base model — you pay for one 14B model in VRAM, not 26 separate models.

---

Both Projects Are Open Source

• beaudamore/biblical — data pipeline, training notebooks, DPO generation, evaluation tooling
• beaudamore/circle-of-speakers-pipeline — pipeline blueprint, all five circle shims, prompt library

The live deployment is at chat.crossandfaith.com. Sign in with Google, select the Biblical Circle, and ask a question.

If you are building something that needs genuinely differentiated voices — for clients, for a product, or as part of a larger AI stack — I am available for consulting and deployment work.

Book an intake call