Engineering
Beyond AI That Speaks Well: Making Kanana-o Speak the Way Users Want
2026년 8월 4일
원문에서 보기 ↗Hello, we are Martin (Daejin Jo), Abigail (Heayoung Park), and Edwin (Wooyoung Kang) from Kanana, the team responsible for developing AI models at Kakao. Our team focuses on building unified multimodal language models that can see, hear, and speak like humans.
Since releasing Korea’s first unified multimodal language model last May [https://tech.kakao.com/posts/702], we have continued to improve the model to provide a better user experience in real-world services [https://tech.kakao.com/posts/802].
As part of this effort, we focused on improving Kanana-o’s speech generation capabilities in two main directions.
-
Generating Speech More Efficiently
To provide natural voice conversations in real-world services, speech quality alone is not enough. Generation speed and computational efficiency are also important. This requires representing speech with shorter and more efficient token sequences, while also simplifying the decoding process that converts generated tokens into actual audio.
-
Moving Beyond “Speaking Naturally Like a Human” to “Speaking the Way the User Wants”
Users have different preferences for speaking speed, style, and volume. They also expect to adjust these characteristics easily depending on their surroundings or purpose. The model must therefore follow the user’s speaking-style instructions accurately while maintaining natural speech quality.
Achieving these two goals requires two key technologies to work together:
- Speech representation technology: A method for representing both the semantic content and acoustic properties of speech in a form that a language model can efficiently learn and generate
- Instruction-following training technology: A method for carefully training the model to reliably follow a wide range of speaking-style instructions
To address the first challenge, we integrated LM-SPT [1], a language-model aligned speech tokenizer developed by Kakao, into Kanana-o. LM-SPT represents speech at a lower frame rate than previous approaches, reducing the length of the token sequences that the Voice Token LM must generate. It also uses a simpler decoding architecture, improving the efficiency of speech reconstruction.
In addition to semantic information, LM-SPT encodes a wide range of acoustic characteristics into tokens. This gives the language model a foundation for directly generating and controlling fine-grained speech properties.
We also enhanced Kanana-o’s speech generation module using Online Reinforcement Learning. This allows the model to better follow a variety of user-provided speaking-style instructions while preserving speech quality and naturalness.
In this post, we discuss the challenges we considered while making Kanana-o generate speech more efficiently and speak in the way users want. We then explain how we addressed these challenges using a new speech tokenizer and Online Reinforcement Learning.
Toward More Efficient and Instruction-Following Speech Generation
How Kanana-o Generates Speech as Tokens
The previously released version of Kanana-o used a speech tokenizer that represented audio as 25 discrete tokens per second. Based on the response generated by the text language model and the conversation context, the Voice Token LM predicts the next speech token one by one.
As depicted in Figure 1, the generated speech tokens are then converted into audio through a two-stage decoding process:
- Token-to-Mel: Converts the speech tokens into a Mel spectrogram.
- Mel-to-Waveform: Reconstructs the final waveform from the Mel spectrogram.
Limitations of the Previous Speech Tokenizer
The previous speech tokenizer used in Kanana-o could compress speech into a relatively small number of tokens while still representing the spoken content reliably. However, further improvements were needed in two areas to provide more precise control over speech generation and improve efficiency in real-world services.
1. A Representation That Makes Acoustic Features Difficult to Control Directly
Human speech contains not only linguistic content, but also many acoustic features such as timbre, pitch, intonation, speaking speed, and emotion.
The previous tokenizer was effective at representing the linguistic and semantic content of speech, but it had limitations in capturing these fine-grained acoustic details. As a result, even when the multimodal language model generated speech tokens, it was difficult to directly control properties such as “speak faster,” “use a lower voice,” or “speak in a whisper” at the token level.
To generate richer and more detailed speech, we needed a tokenizer that could represent both the spoken content and a wide range of acoustic characteristics in a structured way.
2. Long Token Sequences and a Complex Decoding Process
The previous tokenizer represented speech at a rate of 25 token frames per second. As the speech became longer, the Voice Token LM had to generate more tokens sequentially. This directly increased both generation time and computational cost.
In addition, because the previous tokenizer mainly encoded semantic information, the Token-to-Mel model generally required an iterative denoising based multi-step inference model such as diffusion or flow matching.
The generated speech tokens also had to pass through a two-stage decoding pipeline consisting of a Token-to-Mel model followed by a Mel-to-Waveform model before they could be reconstructed as audio.
This architecture made the decoding pipeline more complex and could make it harder to achieve the fast response times required for real-time services.
LM-SPT
To address the limitations discussed above, we developed our own speech tokenizer at Kakao, LM-SPT (LM-aligned Speech Tokenizer), a speech tokenizer designed to work effectively with language models. LM-SPT compresses speech into tokens at 12.5 frames per second, half the frame rate of the previous tokenizer, while preserving both spoken content and fine-grained acoustic characteristics.
Architecture
To speak in the way a user requests, a speech-language model (SLM) must generate not only the content of an utterance but also its acoustic characteristics. However, when these two types of information are mixed together in a single token without a clear separation, the same spoken content may be represented by different tokens depending on the speaker or intonation.
This makes it harder for the language model to predict spoken content reliably from the conversation context. It also becomes difficult to adjust only a specific acoustic property.
For effective integration with a language model, both semantic and acoustic information must therefore be preserved, while their roles remain clearly separated.
- Semantic Speech Tokens: Represent the spoken content corresponding to the text and conversation context.
- Acoustic Speech Tokens: Capture the fine-grained characteristics of the voice, such as speaker timbre, intonation, pitch, and speaking speed.
With this structure, the language model can learn both spoken content and acoustic characteristics while generating each of them in a more stable and fine-grained way.
Separating semantic and acoustic information in this manner has become a common approach in speech tokenizers for recent SLMs. SpeechTokenizer [2] and Mimi, the speech tokenizer used in Moshi [3], both use separate tokens for semantic and acoustic information. They also share a common training strategy: distilling semantic representations from a pretrained speech model to learn tokens that are easier for a language model to process.
LM-SPT also represents speech using two types of tokens: semantic speech tokens and acoustic speech tokens. Its architecture consists of two encoders, one for semantic information and one for acoustic information, as well as a Split RVQ (Residual Vector Quantization) structure with one semantic codebook and multiple acoustic codebooks.
Each codebook is a collection of discrete vectors that represent speech features. For a given input feature, the codebook selects the closest vector and converts it into a token.
Training Method
The semantic codebook, shown as VQ in Figure 2, first represents the core semantic content of the utterance as tokens. Multiple acoustic codebooks, shown as RVQ in Figure 2, then progressively encode acoustic information that was not fully captured in the earlier stages. The speech tokenizer is trained to reconstruct the original speech using both types of tokens.
However, when the tokenizer is trained only for speech reconstruction, there is no guarantee that the semantic and acoustic codebooks will clearly separate their respective information. As the model learns to reconstruct speech more accurately, acoustic properties such as speaker timbre and intonation may become mixed into the semantic tokens. Conversely, the spoken content may be distributed across multiple acoustic tokens.
In order for the semantic-acoustic disentanglement, previous speech tokenizers such as SpeechTokenizer and Mimi mainly use a distillation approach that directly aligns semantic-token features with features extracted by pretrained self-supervised speech models such as HuBERT [6] and WavLM [7] at each time step.
This approach is effective for learning semantically rich tokens, but it has several limitations.
- Mismatch in high-level semantics: The self-supervised models used as distillation teachers mainly capture phonetic information and pronunciation patterns. Their representations may therefore not be fully aligned with the high-level semantic representations used by language models.
- Information loss from matching temporal resolution: To compare features at each time step, the teacher model and the tokenizer must operate at the same or a similar temporal resolution. Because the teacher model usually has a higher frame rate, its representations must be averaged or compressed to match the tokenizer. Important semantic information may be diluted during this process, and the problem becomes more severe as the tokenizer’s frame rate decreases.
To better align semantic tokens with language-model-level semantic representations, LM-SPT uses a pretrained speech encoder suitable for integration with language models, such as Whisper [8]. Instead of directly comparing the teacher and semantic-token features at each time step, we developed and applied a method called Semantic Speech-Resynthesis Distillation.
More specifically, LM-SPT resynthesizes speech using only the semantic tokens. The original and resynthesized speech are then passed through a speech encoder that is well aligned with language models, and LM-SPT is trained to make the semantic representations extracted from the two audio samples similar.
In other words, the semantic tokens are not required to exactly reproduce the teacher feature corresponding to each time step. Instead, they are trained to preserve the same spoken content as the original speech, even under strong compression. This allows the tokens to capture semantic information suitable for language models without requiring artificial alignment between different frame rates.
When converting real speech into tokens or reconstructing speech, LM-SPT does not require a separate, computationally expensive pretrained speech encoder. It only uses lightweight speech encoders that are initialized from scratch and trained as part of the tokenizer.
During final decoding, the semantic and acoustic tokens are used together to reconstruct the waveform directly, without passing through an intermediate representation such as a mel-spectrogram.
As a result, LM-SPT provides a foundation for the Voice Token LM to learn and generate fine-grained speech characteristics at the token level by structurally separating semantic and acoustic information.
Its low frame rate of 12.5 Hz also reduces the number of sequential time steps generated by the Voice Token LM by half. In addition, a lightweight single decoder reconstructs the waveform directly, making the overall speech generation pipeline simpler and more efficient.
Performance Evaluation
The performance of a speech tokenizer such as LM-SPT is generally evaluated from two perspectives:
- Codec performance: Measures the quality of speech reconstructed after converting the input audio into discrete tokens.
- Downstream performance: Measures speech understanding and generation capabilities when the tokenizer is integrated with a Speech-Language Model (SLM).
Compared with other semantic-enhanced speech tokenizers, LM-SPT achieved competitive performance as a speech codec. (Please refer to [1] for further details).
To build a SLM based on a speech tokenizer such as LM-SPT, the language model must be able to accurately understand the spoken content contained in speech tokens. At the same time, it must also be able to synthesize accurate speech from the speech tokens it generates.
To evaluate these capabilities, we combined each speech tokenizer with a language model and experimented on two tasks: Automatic Speech Recognition (ASR) for evaluating speech understanding, and Text-to-Speech (TTS) for evaluating speech generation. To ensure a fair comparison, we applied several recent speech tokenizers to the same SLM under the same training setup.
-
In the ASR experiments, input speech was converted into discrete tokens using each tokenizer. The language model then received these tokens and was trained to generate the corresponding text. We evaluated the models on Korean KSponSpeech and English LibriSpeech under both clean and noisy conditions. Korean performance was measured using Character Error Rate (CER), while English performance was measured using Word Error Rate (WER). For both metrics, a lower score means that the language model understood the content of the speech tokens more accurately.
-
In the TTS experiments, the model received text and speaker information and was trained to generate speech tokens from each tokenizer. We used a zero-shot TTS setting in which speaker embeddings were provided as conditions. For tokenizers with multiple codebooks, we applied the same output module for each codebook and the same one-step delay pattern. To ensure a fair comparison, all tokenizer-based models were given the same speaker reference when synthesizing speech.
| Encoder Params. ↓ | Korean-ASR | Korean-ASR | English-ASR | English-ASR | Zero-shot TTS | Zero-shot TTS | |
|---|---|---|---|---|---|---|---|
| Encoder Params. ↓ | clean (CER↓) | noisy (CER↓) | clean (WER↓) | noisy (WER↓) | Korean (CER↓) | English (WER↓) | |
| CosyVoice2 | 124M | 39.82 | 42.18 | 35.27 | 45.61 | 9.98 | 3.08 |
| Mimi | 38M | 13.65 | 15.2 | 3.82 | 13.65 | 11.03 | 3.29 |
| DualCodec | 622M | 12.92 | 12.93 | 3.95 | 12.13 | 10.37 | 4.62 |
| LM-SPT | 32M | 11.91 | 11.37 | 3.39 | 11.21 | 6.64 | 1.48 |
< Table 1. Comparison of speech understanding and generation performance across SLMs. >
Table 1 compares Korean and English ASR and TTS performance across SLMs built with several recent speech tokenizers. Here, LM-SPT achieved the lowest error rates in every ASR evaluation, covering both Korean and English as well as clean and noisy conditions. It also achieved the lowest error rates in speech generation.
In addition, based on the encoder parameter counts shown in the table, LM-SPT used the smallest encoder among the compared models, with only 32 million parameters. Despite using a relatively lightweight architecture that does not depend on a large pretrained speech encoder, unlike DualCodec [4], LM-SPT achieved the best results in both speech understanding and generation.
Human Evaluation for Zero-Shot TTS
Automatic metrics such as CER and WER can measure how accurately generated speech conveys the intended sentence. However, they do not show whether the speech sounds natural to human listeners or how well it reproduces the target speaker’s voice.
For this reason, TTS systems are commonly evaluated through both automatic metrics and subjective listening tests. We evaluated Korean and English speech generated in a zero-shot TTS setting, where the model synthesized speech using speaker conditions that were not seen during training.
We used the following two metrics:
- NMOS (Naturalness Mean Opinion Score): Measures how natural the generated speech sounds and how closely it resembles real human speech.
- SMOS (Similarity Mean Opinion Score): Measures how similar the generated speech sounds to the target speaker’s voice.
Both metrics are rated on a scale from 1 to 5. A higher score indicates greater naturalness or speaker similarity. For each evaluation, we randomly selected the same set of 15 sentences and evaluated the speech synthesized by each model. 15 experts listened to the samples and rated them.
| Korean NMOS ↑ | Korean SMOS ↑ | English NMOS ↑ | English SMOS ↑ | |
|---|---|---|---|---|
| CosyVoice2 | 3.22 | 2.73 | 2.84 | 2.95 |
| Mimi | 3.10 | 3.38 | 2,96 | 3.73 |
| DualCodec | 3.62 | 3.80 | 3.54 | 3.67 |
| LM-SPT | 3.87 | 3.79 | 3.89 | 3.91 |
< Table 2. Human evaluation results. >
As shown in Table 2, the SLM using LM-SPT achieved the highest NMOS scores in both Korean and English. Among the compared speech tokenizers, LM-SPT was therefore rated as producing the most natural speech.
For SMOS, which measures speaker similarity, LM-SPT achieved the highest score in English. In Korean, it achieved a score comparable to DualCodec, which recorded the highest result.
Decoding Speed Comparison
To evaluate the efficiency of each speech tokenizer, we compared the size and speed of the decoder used to reconstruct the final waveform from generated speech tokens.
Decoding speed was measured using the Real-Time Factor (RTF). RTF is calculated by dividing the time required for processing by the duration of the generated audio. A lower value means that the decoder reconstructs speech more quickly.
For example, if generating 10 seconds of speech results in an RTF of 0.1, the decoding process takes approximately one second.
| Decoder Params. ↓ | RTF ↓ | |
|---|---|---|
| CosyVoice2 | 133M | 0.131 |
| Mimi | 40M | 0.004 |
| DualCodec | 53M | 0.006 |
| LM-SPT | 40M | 0.005 |
< Table 3. Comparison of decoding speed. >
CosyVoice2 [5] uses a two-stage decoding architecture. It first converts speech tokens into a mel-spectrogram and then reconstructs the waveform from the mel-spectrogram. This requires decoding modules with a total of 133M parameters and results in an RTF of 0.131.
In contrast, LM-SPT uses a single-stage decoder that directly reconstructs the waveform from semantic and acoustic tokens, without an intermediate mel-spectrogram.
The LM-SPT decoder has 40M parameters, making it approximately 69% smaller than the CosyVoice2 decoding modules. It also achieved an RTF of 0.005. Under the same experimental conditions, this corresponds to decoding that is approximately 26 times faster than CosyVoice2.
LM-SPT also showed a decoding speed similar to Mimi, which likewise uses a single decoder.
Kanana-o with LM-SPT
Figure 3 shows the speech generation architecture of Kanana-o based on LM-SPT:
- Context processing: Kanana processes the user input and conversation context to produce LLM embeddings.
- Token generation: The Voice Token LM generates semantic and acoustic tokens based on the LLM embeddings and speaker information.
- Speech reconstruction: The LM-SPT decoder directly reconstructs the final speech waveform from the generated tokens.
LM-SPT represents speech at 12.5 frames per second. However, each frame does not contain a single token. Instead, it consists of one semantic token and multiple acoustic tokens. Therefore, to generate the final speech, the Voice Token LM must predict the tokens for all codebooks at every frame.
The simplest approach would be to serialize the semantic and acoustic tokens into one long sequence and generate them one by one. However, this would increase the sequence length in proportion to the number of codebooks. When each frame contains multiple tokens, the actual number of generation steps grows by the number of codebooks, even though the frame rate is only 12.5 Hz. This makes it difficult to fully benefit from the shorter sequence length provided by LM-SPT.
To address this issue, we built a Multi-Codebook Voice Token LM. Instead of expanding all codebook tokens into a single long sequence, the model uses a separate output module for each codebook. At every decoding step, it predicts one semantic token stream and multiple acoustic token streams in parallel.
The gray, yellow, and pink token rows in Figure 3 represent the output sequences for the semantic token and the different acoustic tokens. This design allows the model to represent speech in detail using multiple codebooks, while keeping the overall autoregressive generation process aligned with the 12.5 Hz time axis.
However, the acoustic codebooks in LM-SPT use an RVQ structure. Tokens from later codebooks progressively encode information that was not captured by earlier codebooks. This means that the acoustic tokens are not fully independent, and each later token depends on the tokens generated in the previous codebooks.
If all codebook tokens are predicted at exactly the same time without any ordering, the model may fail to capture these dependencies, which can reduce prediction accuracy. To preserve the relationships between codebooks while still predicting multiple token streams in parallel, we apply a one-step delay pattern to each token sequence.
The semantic token is generated first, and each acoustic token stream is delayed by one additional step relative to the previous codebook. This allows a token from a later codebook to use information from the earlier codebook for the same speech frame.
This approach preserves the progressive information refinement of RVQ while allowing tokens from multiple codebooks to be generated in parallel. After generation, the delays are removed, and the semantic and acoustic token sequences are realigned by frame before being passed to the LM-SPT decoder.
The LM-SPT decoder then uses the tokens from all codebooks together to reconstruct the final waveform directly, without an intermediate representation such as a mel-spectrogram.
As a result, the Multi-Codebook Voice Token LM can use the rich semantic and acoustic representations provided by LM-SPT without causing the excessive sequence growth that would occur if all tokens were generated serially. This allows Kanana-o to directly generate more fine-grained speech information while also improving the efficiency of the overall speech generation process.
Training the Model to Follow Speaking-style Instructions
By applying LM-SPT, Kanana-o became able to represent and generate not only spoken content but also detailed acoustic properties such as timbre, intonation, and speaking speed as tokens. However, having this expressive capability does not automatically mean that the model can accurately follow a user’s instructions.
For example, when a user asks the model to “read this very quickly,” the model should do more than simply speak slightly faster than usual. It must also understand and express the difference in intensity implied by “very quickly.” At the same time, when adjusting speaking speed or volume, the model must preserve pronunciation accuracy, speaker identity, and natural speech quality.
To teach the model this capability, we applied a preference alignment method developed in-house called Counterfactual Preference Alignment. The key idea is to create pairs of speech samples that differ in how well they follow a speaking-style instruction, while keeping all other conditions the same. The model is then trained to distinguish which sample better matches the user’s request.
This approach applies the training framework of Reinforcement Learning from Human Feedback (RLHF) which is widely used for language model alignment, to a speech generation model such as Kanana-o.
In a typical RLHF pipeline, a preference dataset is created by comparing two responses generated from the same input and identifying which response has higher quality. A Reward Model (RM) is then trained to assign a higher score to the preferred response. Finally, the policy model is optimized through reinforcement learning so that its outputs receive higher scores from the RM.
To train a RM that can evaluate whether speaking-style instructions have been followed, the preference dataset must be designed carefully. If two speech samples differ not only in speaking style but also in speaker identity, sentence content, or audio quality, the RM may learn to prefer the sample that sounds more natural or expressive rather than the one that follows the user’s instruction more accurately.
If the RM learns incorrect cues such as speaker characteristics or the mood of a sentence, these biases may become even stronger during reinforcement learning.
For example, suppose one sample is generated using the voice of a more expressive speaker. The RM may assign that sample a higher score regardless of whether it follows the speaking-style instruction. Similarly, a sentence such as “This is a secret” naturally suggests a quiet or whispering voice. The model may therefore judge the speech based on the mood implied by the sentence rather than the user’s explicit instruction.
Another important challenge is distinguishing different levels of instruction intensity. If similar instructions such as “speak quickly” and “speak very quickly” are treated as the same category during training, the RM may fail to recognize the subtle difference in speed and assign similar scores to both samples.
In this case, even after reinforcement learning, the model may still struggle to precisely control the strength of the expression requested by the user.
Building a Counterfactual Preference Dataset
A conventional preference dataset compares two speech candidates generated from the same instruction and assigns a preference to the sample that is better overall. For example, evaluators may listen to two samples generated from the same speaking-style instruction and sentence, then choose the preferred sample based on factors such as naturalness, audio quality, and pronunciation accuracy.
This approach is effective for improving overall speech quality, but it is difficult to identify exactly why one sample was preferred over the other. This becomes particularly problematic when the goal is to improve instruction-following ability. A RM may prefer the voice of a more expressive speaker rather than the sample that more accurately follows the given instruction.
Unlike conventional preference data, our counterfactual preference dataset intentionally changes only the speech property being compared, while keeping all other conditions, such as the speaker and sentence, the same.
The key idea is to compare which speech sample is more appropriate for the current speaking-style instruction and define the preference relationship based on that instruction.
For example, consider three speech samples generated under the following conditions:
- A: Generated with the instruction, “Read the following sentence very quickly.”
- B: Generated with the instruction, “Read the following sentence quickly.”
- C: Generated with the instruction, “Read the following sentence.”
When the three samples are evaluated against the instruction “Read the following sentence very quickly,” sample A best matches the requested condition. We can therefore construct the preference pairs A > B and A > C.
For the instruction “Read the following sentence quickly,” sample B is more appropriate than either A, which is excessively fast, or C, which uses a normal speaking speed. We can therefore define B > A and B > C.
Similarly, when the reference instruction does not request any change in speaking speed, we can construct preference pairs such as C > A and C > B.
By combining one set of speech candidates with different speaking-style instructions, we can expand it into multiple pairwise preference examples without collecting additional speech samples.
Most importantly, even the preference between the same two speech samples may change depending on the instruction used as the evaluation criterion. This encourages the RM to learn how well a speech sample matches a given instruction, rather than becoming biased toward the absolute characteristics of a particular voice or the content of the sentence.
A Reward Model for Evaluating Speaking-Style Instruction Following
Using our Korean counterfactual preference dataset, we trained a RM to determine which of two speech samples follows the user’s instruction more accurately.
The RM is built on Kanana-a, our pretrained SLM (Kanana-a). It takes a speaking-style instruction, the sentence to be spoken, and the generated speech as input, and outputs a score indicating how well the speech reflects the instruction.
Instead of assigning an absolute target score to each sample, we use Bradley–Terry pairwise preference modeling. The model is trained so that the preferred speech sample receives a higher score than the non-preferred one.
This allows us to use a single reward signal to evaluate how well the generated speech follows a wide range of instructions, including emotion, speaking style, speed, and volume.
To evaluate the effectiveness of our preference dataset, we built an in-house benchmark for the RM. We created 1,000 evaluation preference pairs using speech samples with sentences, speaking-style instructions, and speakers that were not included in the training data. Each pair was reviewed by at least three human evaluators.
As shown in Table 4, the RM trained on the counterfactual preference dataset achieved substantially better generalization performance than the model trained on a conventional preference dataset.
| Dataset | Reward Model Classification Accuracy ↑ |
|---|---|
| Conventional preference dataset | 79.1% |
| Counterfactual preference dataset | 86.0% |
< Table 4. Reward model classification performance by preference dataset. >
Multi-Objective Online Reinforcement Learning for Balancing Multiple Quality Criteria
The simplest way to use the preference dataset described above is to directly train the model from predefined preference pairs, as in Direct Preference Optimization (DPO) [12]. The model can be trained to increase the probability of generating preferred speech and decrease the probability of generating non-preferred speech. This type of training is relatively stable and efficient.
However, such offline preference optimization methods generally use only a fixed dataset prepared before training. As a result, it is difficult to examine the strengths and weaknesses of the speech newly generated by the current model and feed those results back into training. In particular, as the model changes during training, it may produce new types of failure that are not sufficiently covered by the predefined preference dataset.
We therefore applied online reinforcement learning (RL) [9, 10] to the speech generation model. In this approach, the current model generates speech, the generated results are evaluated, and the evaluation signals are then used to update the model.
However, optimizing the model only to maximize the RM score can introduce new problems. While trying to emphasize a speaking-style instruction, the model may mispronounce some words, alter the target speaker’s voice, or reduce the overall audio quality.
For example, if the model is optimized only for a reward that favors fast speech, it may increase the speed too much and begin to omit syllables. Because the RM is trained only from preference relationships, it may not fully detect these types of quality degradation.
Instead of relying on a single score predicted by the RM, we therefore applied Multi-objective RL that jointly considers the following four objectives:
- Reward Model score: How accurately does the generated speech follow the speaking-style instruction?
- Speech error rate: Does the model read the given sentence correctly without omitting or mispronouncing any words?
- Speaker similarity: Does the generated speech preserve the target speaker’s distinctive voice characteristics?
- Audio quality score: Does the generated speech sound natural, clear, and free of noise?
By optimizing these objectives together, Kanana-o avoids becoming overly focused on a single evaluation metric. It learns to follow the user’s requested speaking style while preserving audio quality, pronunciation accuracy, and the characteristics of the target speaker.
Performance Evaluation
To evaluate how well the model follows a user’s speaking-style instructions, we built a Korean TTS benchmark with natural-language instructions based on the evaluation framework of InstructTTSEval [11].
During evaluation, the generated speech and its corresponding speaking-style instruction are provided together to a multimodal language model with speech-understanding capabilities. We use an LLM(Gemini)-as-a-Judge approach to determine whether the generated speech matches the instruction.
The evaluation model first analyzes the main speaking characteristics of the generated audio. It returns True when properties such as speaking speed, volume, pitch, and emotion are generally consistent with the instruction. It returns False when one or more important characteristics clearly conflict with the instruction, or when the overall speaking style does not match the request.
The final score is calculated as the percentage of evaluation samples judged to have followed the instruction correctly. This allows us to automatically evaluate complex natural-language speaking-style instructions that are difficult to measure using individual acoustic metrics alone.
As shown in Table 5, Multi-objective online reinforcement learning produced a much larger improvement than DPO on the Korean InstructTTSEval benchmark. The results also show that the Kanana-o achieved speaking-style instruction-following performance comparable to state-of-the-art TTS models from other providers.
| Model | Korean InstructTTSEval Score |
|---|---|
| GPT-4o-mini-tts | 91.10 |
| Gemini-2.5-flash-preview-tts | 95.38 |
| Kanana-o (initial checkpoint) | 73.90 |
| Kanana-o (after DPO) | 81.30 |
| Kanana-o (after Multi-objective online RL) | 94.50 |
< Table 5. Comparison of Korean InstructTTSEval performance. >
Examples of Speech Synthesis with Speaking-Style Instructions
Let us look at several examples that demonstrate the improved instruction-following ability of Kanana-o with LM-SPT. In the examples below, Kanana-o follows both simple and complex natural-language instructions to control a wide range of acoustic properties in the generated speech.
Sentence to be spoken: “카카오는 단순한 텍스트 생성을 넘어 AI가 스스로 단계별 실행 계획을 수립하고, 필요한 기능을 호출해 과업을 완수하는 ‘종합 실행 역량’ 검증에 집중하고 있습니다.”
Instruction Type: “None”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read the sentence | 음성 듣기 | 음성 듣기 |
Instruction Type: “Volume”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it quietly | 음성 듣기 | 음성 듣기 |
| Read it loudly | 음성 듣기 | 음성 듣기 |
| Read it very quietly | 음성 듣기 | 음성 듣기 |
| Read it in a very quiet whisper | 음성 듣기 | 음성 듣기 |
Instruction Type: “Speed”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it quickly | 음성 듣기 | 음성 듣기 |
| Read it extremely quickly | 음성 듣기 | 음성 듣기 |
| Read it extremely fast, almost frantically | 음성 듣기 | 음성 듣기 |
| Read it a little slowly | 음성 듣기 | 음성 듣기 |
| Read it slowly | 음성 듣기 | 음성 듣기 |
| Read it very slowly | 음성 듣기 | 음성 듣기 |
Instruction Type: “Pitch”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it in a higher-pitched voice | 음성 듣기 | 음성 듣기 |
| Raise your voice as high as you can | 음성 듣기 | 음성 듣기 |
| Read it in a low-pitched voice | 음성 듣기 | 음성 듣기 |
| Read it in a very low-pitched voice | 음성 듣기 | 음성 듣기 |
Instruction Type: “Accent and Delivery”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it in a Jeolla-do dialect | 음성 듣기 | 음성 듣기 |
| Read it in a Gyeongsang-do dialect | 음성 듣기 | 음성 듣기 |
| Read it as if you were rapping | 음성 듣기 | 음성 듣기 |
Instruction Type: “Emotion”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it in a bright and cheerful voice | 음성 듣기 | 음성 듣기 |
| Read it brightly and cheerfully, like a child | 음성 듣기 | 음성 듣기 |
| Read it in a sad voice | 음성 듣기 | 음성 듣기 |
| Read it in a low, weak voice with a sorrowful tone | 음성 듣기 | 음성 듣기 |
| Read it as if you were angry | 음성 듣기 | 음성 듣기 |
| Read it like someone who is sleepy and cannot be bothered with anything | 음성 듣기 | 음성 듣기 |
Instruction Type: “Role”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it like a sports commentator | 음성 듣기 | 음성 듣기 |
| Read it like a news anchor | 음성 듣기 | 음성 듣기 |
| Read it loudly and expressively, as if you were reading a storybook aloud | 음성 듣기 | 음성 듣기 |
Instruction Type: “Complex”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it quickly in a whisper | 음성 듣기 | 음성 듣기 |
| Read it slowly in a higher-pitched voice | 음성 듣기 | 음성 듣기 |
| Read it slowly in a higher-pitched voice, like a child | 음성 듣기 | 음성 듣기 |
| Read it quickly in a lower-pitched voice, like a child | 음성 듣기 | 음성 듣기 |
| Read it quickly in a higher-pitched voice, like a child | 음성 듣기 | 음성 듣기 |
| Read it quickly in a low-pitched, sad voice | 음성 듣기 | 음성 듣기 |
| Read it quickly in a low-pitched, sad voice, like a child | 음성 듣기 | 음성 듣기 |
Interestingly, although both the preference dataset and the online reinforcement learning data consisted only of Korean speech, the learned speech control capabilities also worked for English speech generation.
One possible reason is that acoustic properties such as speaking speed, volume, pitch, and emotion are not fully tied to the linguistic content of a specific language. Instead, they can serve as shared control factors across different languages.
LM-SPT is particularly well suited to this transfer because it structurally separates semantic tokens, which represent spoken content, from acoustic tokens, which represent detailed speech characteristics. As a result, even when the model learns the relationship between speaking-style instructions and acoustic changes using Korean data, the learned control behavior does not need to remain tied to specific Korean expressions. It can also appear when generating speech in other languages, including English.
These results suggest that a speech representation with a clear separation between semantic and acoustic information can help transfer speech control capabilities learned in one language to another.
The examples below show that Kanana-o can also generate English speech that follows the given speaking-style instructions.
Sentence to be spoken: “With the rapid progress of speech language models, discrete speech tokens have emerged as a core interface between speech and text, enabling unified modeling across modalities.”
Instruction Type: “None”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read the sentence | 음성 듣기 | 음성 듣기 |
Instruction Type: “Volume”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it in a very quiet whisper | 음성 듣기 | 음성 듣기 |
| Read it loudly | 음성 듣기 | 음성 듣기 |
Instruction Type: “Speed”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it quickly | 음성 듣기 | 음성 듣기 |
| Read it extremely fast, almost | 음성 듣기 | 음성 듣기 |
| Read it a little slowly | 음성 듣기 | 음성 듣기 |
| Read it very slowly | 음성 듣기 | 음성 듣기 |
Instruction Type: “Pitch”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it in a higher-pitched voice | 음성 듣기 | 음성 듣기 |
| Read it in a low-pitched voice | 음성 듣기 | 음성 듣기 |
Instruction Type: “Accent and Delivery”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it in a Korean dialect | 음성 듣기 | 음성 듣기 |
| Read it as if you were rapping | 음성 듣기 | 음성 듣기 |
Instruction Type: “Emotion”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it brightly and cheerfully, like a child | 음성 듣기 | 음성 듣기 |
| Read it in a sad voice | 음성 듣기 | 음성 듣기 |
| Read it in a low, weak voice with a sorrowful tone | 음성 듣기 | 음성 듣기 |
| Read it as if you were angry | 음성 듣기 | 음성 듣기 |
| Read it like someone who is sleepy and cannot be bothered with anything | 음성 듣기 | 음성 듣기 |
Instruction Type: “Role”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it like a sports commentator | 음성 듣기 | 음성 듣기 |
| Read it loudly and expressively, as if you were reading a storybook aloud | 음성 듣기 | 음성 듣기 |
Instruction Type: “Complex”
| Speaking-Style Instruction | Speaker 1 | Speaker 2 |
|---|---|---|
| Read it quickly in a whisper | 음성 듣기 | 음성 듣기 |
| Read it slowly in a higher-pitched voice | 음성 듣기 | 음성 듣기 |
| Read it very quickly in an extremely deep, low-pitched voice | 음성 듣기 | 음성 듣기 |
| Read it very quickly in an extremely deep, low-pitched voice, like a sports commentator | 음성 듣기 | 음성 듣기 |
Future Directions
1. A More Advanced Speech Tokenizer
Recent research on speech tokenizers has increasingly focused on learning unified speech token representations that can be effectively integrated with speech-language models and used for both speech understanding and generation.
Existing SLMs often use different representations for the two tasks: encoder representations for understanding speech and token representations for generating speech. However, if a single speech tokenizer can support both roles, the model can understand and generate speech within the same representation space.
This unified architecture offers several clear benefits:
- Representation sharing and cross-task synergy: Semantic and acoustic representations learned through speech understanding and generation can be shared and reused across both tasks. This allows the two tasks to reinforce each other and improve performance together.
- Simpler model architecture: Rather than using different representation systems and separate processing pipelines for speech understanding and generation, the model can be built around a single unified speech representation. This reduces redundant modules and representation-conversion steps, simplifies the overall architecture, and improves implementation and deployment efficiency.
- Better support for real-time and on-device applications: A unified representation reduces redundant computation, memory use, and conversion steps between speech understanding and generation. This helps lower latency in real-time conversations and enables more efficient operation in on-device environments with limited computational resources and memory.
LM-SPT, introduced in this post, achieved strong performance in both speech understanding and generation through its low frame rate and language-model-friendly token representations. However, when considering speech understanding alone, there is still room for improvement compared with dedicated speech encoders such as Whisper [8], which are trained on large-scale datasets specifically for speech recognition.
We plan to further strengthen the semantic representation capabilities of speech tokenizers so that a single tokenizer can approach the performance of specialized models in both speech understanding and generation.
LM-SPT currently represents speech at 12.5 frames per second. In the future, we also plan to achieve a higher compression rate while preserving reconstruction quality, further reducing the number of tokens that the model needs to generate.
2. Supporting Real-Time Speech Applications
Building on LM-SPT’s low frame rate and high computational efficiency, we are expanding it to a wider range of SLM applications that understand and generate speech in real time.
In particular, we aim to provide natural and uninterrupted voice interactions in latency-sensitive services such as full-duplex voice assistants, real-time speech recognition, and real-time speech translation.
3. Better Nonverbal Expression and Fine-Grained Speech Control
Today’s SLMs are moving beyond simply reading text naturally. They are increasingly expected to express a wide range of nonverbal behaviors, much like humans do.
We plan to improve the model so that it can naturally generate nonverbal vocalizations such as laughter, sighs, and exclamations. For example, when a tag such as [laughter] is inserted into a sentence, the Voice Token LM should generate natural laughter by considering both the position of the tag and the surrounding context.
We also plan to support more precise control over emotion and delivery by allowing users to insert control tags such as [empathetically], [whispering], and [with emphasis] at specific positions in an utterance. Compared with a single natural-language speaking-style instruction applied to the entire sentence, these tags would make it possible to control how different parts of the utterance are expressed in a more fine-grained way.
Closing
In this post, we introduced how Kanana-o has evolved to speak more naturally and express speech in different ways based on the user’s intent. From the speech tokenizer and Voice Token LM to online reinforcement learning, each technology was developed not only to improve model performance, but also to make conversations with users feel more natural and comfortable.
Kakao will continue to advance the core technologies needed to understand and generate speech more efficiently. We also plan to develop next-generation SLMs that support real-time voice interfaces and more precise speech control.
Our goal is to provide a voice experience that feels so natural and convenient that users no longer need to think about the fact that they are talking to AI.
We hope you will continue to follow and support Kanana-o as it evolves.
Contributors
This work was carried out by the following members of the Kanana team:
abigail.r (Heayoung Park), brook.p (Beomhee Park), charlie.robust(Jungwon Choi), daniel.log (Youngjune Kim), edwin.ai (Wooyoung Kang), james.e (Jaemyung Lee), jayden.x (Jongcheol Boo), jayten.ten (Jaeyeol Jeon), jessie.e (Jeehye Lee), johnny.oh (Jaehoon Oh), martin.gale (Daejin Jo), peter.brain (Byungseok Roh), polar.bears (Jihwan Eom), sam.shkim (Saehoon Kim), samuel.brain (Sunghun Kang), and welt.bae (Byeonguk Bae).
We would also like to thank jennie.ee (Jihye Lee) and jeri.s (Heehyun Lee) from the Kanana team for their significant contributions to data collection and preparation.
Acknowledgments
We would like to thank peter.brain (Byungseok Roh) and hunter.jo (Hoon Jo) from the Kanana team for carefully reviewing the entire article.
References
- [1] D. Jo, J. Yun, B. Roh, and S. Kim, “LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization”, IEEE Transactions on Audio, Speech and Language Processing, 2026.
- [2] X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu, “SpeechTokenizer: Unified speech tokenizer for speech language models,” International Conference on Learning Representations, 2024.
- [3] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave et al., “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024.
- [4] J. Li, X. Lin, Z. Li, S. Huang, Y. Wang, C. Wang, Z. Zhan, and Z. Wu, “DualCodec: A low-frame-rate, semantically-enhanced neural audio codec for speech generation,” Interspeech, 2025.
- [5] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang et al., “CosyVoice 2: Scalable streaming speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024.
- [6] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE Transactions on Audio, Speech and Language Processing, 2021.
- [7] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, 2022.
- [8] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” International Conference on Machine Learning, 2023.
- [9] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, 2022.
- [10] D. Khatri, L. Madaan, R. Tiwari, R. Bansal, S. S. Duvvuri, M. Zaheer, I. S. Dhillon et al., “The art of scaling reinforcement learning compute for LLMs,” International Conference on Learning Representations, 2026.
- [11] K. Huang, Q. Tu, L. Fan, C. Yang, D. Zhang, S. Li, Z. Fei, Q. Cheng, and X. Qiu, “InstructTTSEval: Benchmarking complex natural-language instruction following in text-to-speech systems,” arXiv preprint arXiv:2506.16381, 2025.
- [12] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model.” Advances in Neural Information Processing Systems, 2023