Soul-Guided Fine-Tuning: Teaching Gemma 4 to Be a Dental Receptionist

The Problem

Frontier LLMs are impressive generalists, but they aren't dental receptionists. When we benchmarked Claude Opus 4.6, Sonnet, GPT-5.4, and Gemini on DentesBench, the best model scored 8.03 out of 10. That's good — but it's not good enough for production. Patients notice when the agent sounds slightly off, slightly robotic, or slightly too eager to offer clinical reassurance.

The issue isn't capability. These models know what a dental receptionist should say. The issue is discipline — consistently choosing the warm-but-boundaried response over the helpful-but-presumptuous one, hundreds of times a day, across every possible patient scenario.

Generic instruction tuning doesn't teach discipline. RLHF with broad human preferences doesn't teach it either, because human raters tend to prefer the warmer response even when it crosses clinical lines. What's needed is domain-specific preference optimization, guided by an explicit specification of what "good" means in this exact context.

The Approach

The soul document

Inspired by constitutional approaches to AI alignment, we wrote a comprehensive character specification for the dental phone agent — what we call its soul document. This isn't a system prompt. It's a detailed document that defines the agent's identity, values, knowledge boundaries, conversation principles, and named failure modes.

This document serves a dual purpose: it guides training by providing the rubric that a judge model uses to evaluate and rank responses, and it defines the scoring criteria for DentesBench evaluation. One document, two functions. The specifics of what it contains — and how it shapes model behavior — are part of our proprietary training methodology.

The self-training loop

The pipeline combines real dental clinic conversations with synthetically generated scenarios, trains the model in two stages (supervised learning followed by preference optimization), then evaluates the result against a fixed benchmark. Weak areas are automatically identified and over-sampled in the next iteration.

The key insight is that the judge model evaluates responses against the soul document on five specific dimensions — the same dimensions used in DentesBench. This means the training signal is directly aligned with the evaluation metric. The model isn't just learning to be "generally helpful" — it's learning to embody a specific, measurable standard of dental reception quality.

Results

Where it excels

Clinical safety: 9.93. This is the dimension we care about most. The fine-tuned model almost never crosses into diagnosis territory. When a patient says "is this normal?", it consistently responds with empathy and a handoff rather than clinical speculation. The soul document's hard boundary on clinical safety was deeply internalized during preference optimization.

Accuracy: 8.49. Higher than every API model we tested. The model doesn't hallucinate costs, availability, or insurance details. When it doesn't know something, it says so.

Brevity: 8.60. Phone-appropriate responses. The model learned that 2-4 sentences is the right length — not the information dumps that generic models tend to produce.

Where it struggles

Natural tone: 7.16. This is the weakest dimension. The model is safe and accurate but still sounds slightly automated. It uses correct phrasing but lacks the natural variance that makes a human receptionist feel human — slightly different wordings each time, conversational fragments, the occasional "let me see" before looking something up.

This is exactly what the next training iteration targets. The self-correcting loop identifies weak dimensions and generates more training data specifically designed to push those scores up — in this case, preference pairs that contrast robotic-but-correct responses with natural-and-correct ones.

What This Means

The cost of a single training iteration is modest — under $250 in total compute and API costs, completable in a day. This makes iterative improvement practical: run an iteration, evaluate, identify weaknesses, run again. Each cycle is targeted, not random. The model gets specifically better at the things it was specifically bad at.

What's Next

This is iteration 1 of an ongoing research effort. The self-training loop is designed to run repeatedly, and we're actively working on several fronts:

• Conversational naturalness. The primary weakness. We're developing preference pairs that specifically target the gap between "correct but robotic" and "correct and human."
• Model distillation. 31B parameters is too large for cost-effective real-time phone calls. We're exploring distillation into smaller architectures that maintain quality at production-viable latency.
• Tool-calling integration. Production agents don't just talk — they book appointments, check schedules, and verify insurance. Maintaining conversational quality while executing structured workflows is a distinct challenge.
• DentesBench v1.0. Expanding the benchmark to 1,000+ scenarios with human evaluation baselines, multilingual support, and tool-calling evaluation.

The broader goal is a model that clears 8/10 on all five soul dimensions simultaneously, with zero critical anti-patterns, at a cost and latency that works for real dental clinics. Iteration 1 proved that domain-specific fine-tuning can surpass frontier API models on a task-specific benchmark. The remaining work is making it consistently excellent across every dimension — and fast enough to answer the phone.