If you have applied a speech-recognition model to a Korean-language service, you may have found that multilingual models such as Whisper are less reliable than expected. If recognition seems particularly poor in meetings full of foreign technical terms or in the speech of multilingual users who mix languages, you are not imagining it.

In Theta One's recent research, models with error rates below 5% on Korean-only or English-only speech showed errors up to 14 times higher when languages were mixed in one utterance. A model that looks service-ready at a 5% error rate can reach 60% in some use cases. 😱

Here is our research on measuring Korean-English code-switching recognition accurately and finding ways to improve it:

• Paper
• Code
• Dataset

---

Hello! I'm Gio, an ML engineer responsible for speech AI research and development at Theta One. 😎

Today I'll introduce our paper, HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition, and briefly describe Theta One AI's speech-recognition technology.

In this study, we:

1. proposed HiKE, a framework for evaluating Korean-English code-switching speech recognition;
2. used it to show that errors in multilingual ASR models, including Whisper, can increase by up to 14 times compared with monolingual speech; and
3. showed that synthetic data can mitigate some of the problem at relatively low cost.

🤔 What is code-switching?

Code-switching (CS) means alternating between languages within a conversation or context. Mixing Korean with English IT jargon, sometimes called "Pangyo-speak" in Korea, is a familiar example:

"Hey, Gio, 오늘 아침에 dev branch 오류 관련해서 issue 남긴 거 봤어요?"
"Hey, Gio, did you see the issue I filed this morning about the dev branch error?"

People who usually speak with others in only one language may encounter this less often. But code-switching is commonplace in fields such as IT and medicine, where foreign terms travel easily, in language learning, and among multilingual speakers who naturally mix languages.

💭 Forms of code-switching​

The pattern varies by language pair. A speaker may substitute a single noun, adjective, or verb (word-level CS), switch several spans within a sentence (phrase-level CS), or use different languages in different sentences (sentence-level CS). Each poses a different challenge for speech recognition.

With sentence-level code-switching, the switch occurs at a sentence boundary and is relatively regular:

"그 부분은 저희 팀원이 조금 더 잘 설명할 수 있을 것 같은데요. Shawn, will you explain this?"
"My teammate can probably explain that part better. Shawn, will you explain this?"

With word- or phrase-level code-switching, the switch can happen anywhere inside a sentence, creating more and less predictable transition points:

"선생님, 'should have'하고 'would have' 차이가 뭔지 전혀 모르겠어요."
"Teacher, I don't understand the difference between 'should have' and 'would have.'"

Phrase-level switching between languages with different grammars, such as Korean and English, can produce structures that are difficult even for people to parse:

"빙빙 돌려말하지 말고 get to the point로 말 해봐. You know what I'm talking about?"
"Stop talking around it and get to the point. You know what I'm talking about?"

For people working with English technical terms, learning English with AI, or speaking two languages naturally, accurate recognition of this mixed speech is essential.

✨ HiKE: A Korean-English code-switching benchmark with deeper analysis

Theta One AI developed HiKE, the first evaluation framework focused on Korean-English code-switching ASR, to assess a capability that had not been studied systematically.

HiKE stands for Hierarchical Korean-English Code-Switching Benchmark. It measures model performance separately on the three forms of code-switching described above. Previously, without an objective way to assess this performance, teams had to collect hard-to-find data themselves or use models whose code-switching accuracy had not been validated. HiKE gives them a benchmark for comparison and improvement.

Realistic speech across different topics​

To test the behavior that services actually need, HiKE collected code-switching speech across eight topics where people commonly mix languages.

We classified the collected 2.2 hours of samples by topic and code-switching type so performance could be measured for realistic situations.

Metrics for code-switching speech recognition​

Monolingual ASR is usually evaluated with character error rate (CER) or word error rate (WER). CER is often used for Korean, with its complex syllable structure and frequent spacing errors; WER suits English, where word boundaries are clearer.

One metric alone can be misleading when Korean and English appear together. We used Mixture Error Rate (MER) and Point of Interest Error Rate (PIER) for a more precise evaluation.

MER counts errors at the character level for Korean and at the word level for English. It measures overall transcription accuracy.

MER = (insertions + deletions + substitutions) / N

PIER measures errors only in areas selected by the evaluator. Around code-switching points, it tells us whether a model handles the transition between languages correctly.

<tag 이번> <tag bug> <tag 는> <tag session> management <tag logic> <tag 에> 문제가 있었어
"There was a problem with the session management logic in this bug."

For example, the tagged words mark the regions immediately around language switches. Using both metrics lets us assess general transcription accuracy and accuracy at the points where languages change.

Evaluation results​

We evaluated a wide range of multilingual ASR models: CTC-based SenseVoice Small, known for a code-switching-robust architecture; the widely used Whisper family; and large-model systems including NVIDIA's Audio Flamingo 3 and OpenAI's GPT-4o. Most performed well on Korean-only and English-only speech, yet MER and PIER errors rose by roughly 3 to 14 times when the languages were mixed.

Many models suffered especially on word- and phrase-level code-switching, which occurs more often in realistic conversation than switching only between complete sentences. Strong monolingual results can therefore conceal a serious decline in user experience when an ASR model reaches a real service.

We separately evaluated our own Theta One STT model (version 2025.07). It achieved lower MER and PIER error rates than even the much larger GPT-4o model.

Theta One STT, which supports Korean-English code-switching and children's speech recognition, is available in the Theta One AI console.

Improving Korean-English recognition with synthetic data​

If most models have error rates too high to use in Korean-English code-switching, how can we improve them?

Collecting enough natural code-switching speech for fine-tuning would be ideal, but this rare data is expensive and laborious to obtain. We instead constructed synthetic sentence-level code-switching data by concatenating Korean-only and English-only speech, which are easier to source, and used it to fine-tune a model.

Evaluation on HiKE showed surprisingly large improvements in MER and PIER even after fine-tuning on synthetic data. It did not match fine-tuning with real code-switching data (a), but simple concatenation can still be useful for a lower-cost proof of concept.

Theta One STT: From code-switching to children's speech​

Training the model is only one step toward using STT in a service. Teams also need metrics that reflect production behavior beyond benchmark scores, model optimization for efficient deployment, a GPU serving platform, and the budget for substantial hosting costs.

Theta One offers a high-performance STT model through an API that supports code-switching and children's speech recognition. We continue working on its accuracy and speed.

Visit Theta One AI's console to try our speech-recognition AI at a reasonable price.

Thank you for reading!