The problem
A service centre handles well over a thousand calls a day, and almost none of them were searchable afterwards. Agents typed notes from memory, quality reviews sampled one call in two hundred, and cloud transcription services were ruled out because the recordings contain names, addresses and payment details.
The brief: transcribe every call accurately in German (and the occasional English one), keep all audio on site, and turn the transcripts into something a team lead can act on.
What we built
- A Whisper-family speech model fine-tuned on 600 hours of consented, anonymised call audio, which halved the word error rate on regional accents, product names and order numbers.
- Streaming inference with speaker diarisation, so the transcript appears while the call is still running and every line is attributed to agent or customer.
- A redaction stage that detects phone numbers, addresses, IBANs and card numbers in the transcript and replaces them before anything is stored.
- A small language model that produces a summary, action items, topics and a per-turn sentiment score for every call; the agent confirms the CRM note with one click.
Results
The fine-tuned model reaches 6.1 % word error rate on held-out calls, down from 14.3 % for the base model, at a real-time factor of 0.18 — one mid-range GPU keeps up with the entire centre. After-call work dropped by around 40 % because notes are drafted before the agent hangs up, and quality reviews now cover every call rather than a sample.
Sentiment trends across topics gave the team its first early-warning signal: a delivery-partner problem showed up in the call data two days before it appeared in the complaint statistics.