Why an accurate video transcript is the foundation of Reelang, and why we run it on Deepgram

Reelang teaches languages from real short-form video, the clips people actually watch, in the language they're trying to learn. Around that we've built things we're proud of: subtitles that quietly point you at the words worth learning first, difficulty scoring, per-clip learner insights. None of it works without one plain thing underneath. We have to know exactly what was said, and exactly when.

That's a speech-to-text problem, and it's the part of our stack we worry about least. We run it on Deepgram, and after a lot of testing across a lot of languages, the transcript quality has been good enough that we've stopped treating it as a risk. If you're a developer weighing speech-to-text for anything beyond clean English studio audio, the rest of this is the honest version of what we found.

The quickest way to show what we mean is one real clip from our Spanish catalog: Luis Díaz's first day at Bayern, a Colombian footballer meeting a German club. Here is what a single Deepgram call knows about it.

{
  "word": "danke",
  "start": 57.585,
  "end": 58.305,
  "confidence": 0.6801,
  "speaker": 1,
  "speaker_confidence": 0.237,
  "punctuated_word": "Danke.",
  "language": "de"
}

We benchmarked the field before we committed

We didn't pick a vendor and hope. We ran the same real-world clips through the obvious candidates: OpenAI Whisper, Deepgram, ElevenLabs, AssemblyAI, and a few open-source/open-weights models. Our audio is the hard kind. It's phones in the street, kitchens, cars, markets, creators talking fast over music, two people talking at once. That's the whole point of Reelang, so a model that only shines on clean audio is no use to us.

Deepgram came out ahead, and the gap was widest exactly where it mattered most: background noise and music. Clips where a creator talks over a backing track, or films in a loud room, are where the other models started dropping and mangling words. Deepgram held together on those. On clean audio in the top 10 languages most of the field was doing fine; on the messy audio that makes up most of the real world, the differences showed up fast.

We also benchmarked against the captions the platforms generate themselves. YouTube and TikTok both auto-caption uploads, and for a while we leaned on those where they existed. The quality is not close. On the same clips, Deepgram's transcript and word timing were far ahead of the native auto-captions, especially once there's an accent, some noise, or a language other than English. That single finding reshaped our pipeline: for most content we transcribe from the audio ourselves rather than trust what the platform stamped on.

What this is all for

Before the pipeline, here's the payoff. This is the Reelang watch experience for a Spanish clip. The caption stays in sync with the speech, you can tap a word to see what it means, and when you swipe to the next clip the words you've already met light up again. Every piece of that, the caption timing, the tappable word, the highlight, is placed from the transcript. Get the transcript wrong and this whole surface quietly breaks.

Telling the model what you already know

If you're searching for good speech-to-text for a specific language, the most useful thing we can pass on isn't a single model name. It's two habits.

First, give the model the language when you already know it. A lot of our clips arrive with the language obvious from the source metadata. When it's clear from the start, we hand that language to Deepgram as a hint instead of letting it auto-detect, and the transcript comes back better. Auto-detect is a fine fallback, but a known language passed in beats it almost every time. It's the same reason we keep a variant like Swiss German (de-CH) separate from standard German: when we already know a clip is Swiss German, saying so gets a cleaner result than treating it as German.

Second, lean on word-level language tags for code-switching. Deepgram tags language per word, not just per clip. So when a Spanish creator drops English words mid-sentence, we can see exactly which words switched, rather than losing them or having them mangled into nearby Spanish. For our highest-volume languages we run in a multilingual mode that expects this, so loan words land as themselves and the per-word tags tell us where the switches happened.

For our use case, this also helps us differentiate truly native in-language content from instructional videos (e.g. a Portuguese creator explaining something in English, then switching back to Portuguese).

We route each clip to one of Deepgram's models based on the language:

The takeaway for anyone evaluating: test your specific language on your real audio, and pass the language in when you know it rather than relying on detection. A vendor's headline number tells you little about the one language you actually care about.

Everything reads from the transcript

It's easy to think of transcription as one feature among many. For us it sits under all of them. Every step downstream reads from the transcript:

• Smart subtitles. We work out how common each word is in the target language, then use that to decide what to highlight, so your attention lands on the vocabulary that pays off first instead of getting lost in a wall of text. That ranking is only as good as the words we think were spoken.
• Difficulty scoring. We estimate how hard a clip is to follow, from the language actually used and how it's spoken. A transcript that drops or mangles words skews that estimate and sends learners the wrong clips.
• Learner insights. The short notes that explain what's going on in a line of speech are generated from the transcript. If the transcript is wrong, the insight comes out fluent, plausible, and false.

Errors at the bottom of a stack don't stay put. One mis-transcribed word doesn't just show up as one wrong subtitle. It skews the frequency ranking, nudges the difficulty score, and can produce a learner insight that reads perfectly and says something untrue. So the bar here isn't "readable." It's accurate enough that we can build on it without checking its work.

The timing matters as much as the words

For us a transcript is text and timing together. Deepgram gives us word-level timestamps, a start and end time for each word with a confidence score attached, and that timing does quiet, load-bearing work.

The obvious use is sync: to highlight one word inside a caption exactly as it's spoken, we need to know when that word starts and ends, not just when the whole line does. To let you tap a word and hear it again, we need its edges.

The less obvious use is that the timing tells us how hard a clip is to follow. How fast someone talks matters, but so does their rhythm: how long they run on before they pause. Research on listening (De Jong and Bosker, 2013) uses a 250-millisecond gap as the mark of a real pause, and the stretch of speech between pauses is a good proxy for the load on a learner's brain. Short bursts with frequent breaks are easy to keep up with. Long unbroken runs are hard even when the individual words are simple, because there's nowhere to catch up.

We can only measure any of that because Deepgram hands us the gaps between words. From the word timings we compute the pace, the mean length of run (how many words go by between pauses), and how densely packed the speech is, and we blend those into a speech-pattern score that we calibrated against 682 human-rated clips. Two real examples from our Spanish catalog:

• A gentle, slow clip runs at about 80 words a minute in short bursts of roughly four words between pauses. Easy to follow.
• A fast influencer parody runs at over 200 words a minute in long runs of dozens of words with barely a break. Hard to follow, and our score says so, even though plenty of the words are common.

Words-per-minute alone would rate those two much closer than they feel. The pause pattern is what separates them, and the pause pattern only exists because the transcript carries accurate per-word timing.

Confidence scores we can act on

Deepgram returns a confidence signal next to the words, and a signal that means something is worth as much as the words themselves. It lets us be honest about which transcripts to trust outright, which to route differently, and which content isn't a good fit at all. A model that's sometimes wrong but knows when it's unsure beats one that's confidently wrong, because we can build guardrails around uncertainty and we can't build them around a lie.

The result: transcription has dropped off our list of things to babysit. We spend our attention on the layers above it, the frequency ranking, the scoring, the insights, the localization, because the layer underneath has earned the room to be left alone.

Boring in the best way

The best thing you can say about a piece of infrastructure is that you forget it's there. Deepgram has mostly earned that. The transcript shows up accurate, on time, with per-word timing and a confidence score we can reason about, and the rest of Reelang gets to treat it as right.

That's not a small thing. It's the difference between building and constantly re-checking whether the ground under you holds. We got to skip the second part, and it shows up as speed everywhere else.

If you're building anything that reasons over what people actually said, and not only shows it back, treat the transcript as the foundation it is and choose your speech-to-text with that in mind. Test it on your messiest audio, in your real languages, in your production mode. We did, and it's one of the calls we've had the least reason to revisit.

Questions, or ideas for what we should build next? We hang out at r/Reelang. Come poke holes in how we do this, ask us anything, or tell us which language-learning features you'd actually find useful. We read all of it.