Speech-Style-Aware Grapheme-to-Phoneme (G2P) Conversion for Expressive Speech Synthesis

Table of Contents

1 Introduction

1.1 Host institution

1.2 Research purpose

1.3 Background

1.3.1 Speech style in speech synthesis

1.3.2 Grapheme-to-Phoneme (G2P) conversion

1.4 Organization of the report

2 Sequence-to-sequence (Seq2Seq) encoder-decoder architecture for speech-style-aware G2P conversion

2.1 An overview of deep learning (DL) and neural networks

2.1.1 Feedfoward neural networks

2.1.2 Recurrent neural networks (RNNs) and long short-term memory (LSTM)

2.2 Sequence-to-sequence (Seq2Seq) encoder-decoder framework

2.3 Attention mechanism in encoder-decoder framework

2.4 Adaptation of Seq2Seq architecture for speech-style-aware G2P conversion

3 Experiments and results

3.1 Preliminary experiments

3.1.1 Experiment setup

3.1.2 Results

3.2 Using short segments

3.2.1 Constructing short segments and model configuration

3.2.2 Results

3.3 Architecture selection

3.4 Impact of word boundaries

3.5 Impact of pre-training

3.6 Evaluation for various emotions

4 Conclusion

A CMUdict for English results

Research Objectives and Topics

The main objective of this research is to investigate the contribution of segmental pronunciation variations to expressivity in speech synthesis for the French language. The research focuses on developing deep neural network models capable of transforming a neutral or canonical pronunciation into a specific pronunciation that corresponds to a target speech style or emotional state, thereby overcoming the limitations of standard neutral speech synthesis.

• Investigation of pronunciation variations (segmental) as a cue for emotional expressivity in French.
• Application of Grapheme-to-Phoneme (G2P) conversion to model specific speech styles and emotions.
• Utilization of sequence-to-sequence encoder-decoder architectures (specifically RNN/LSTM).
• Evaluation of techniques such as short segment construction, word boundary marking, and model pre-training.
• Improvement of Phoneme Error Rate (PER) performance in expressive TTS systems.

Excerpt from the Book

Summary of Chapters

1 Introduction: Provides an overview of the host institution and the research motivation, highlighting the need for more expressive TTS systems beyond neutral speaking styles.

2 Sequence-to-sequence (Seq2Seq) encoder-decoder architecture for speech-style-aware G2P conversion: Reviews the deep learning fundamentals and the specific Seq2Seq encoder-decoder framework used to address the G2P conversion task.

3 Experiments and results: Details the experimental setup using the Caroline corpus, including the investigation of short segments, word boundaries, and the impact of pre-training on model performance.

4 Conclusion: Summarizes the key findings of the project and suggests potential directions for future research to further improve model expressivity and performance.

A CMUdict for English results: Presents the evaluation results obtained by applying the proposed model architecture to the English CMUdict dataset.

Keywords

Deep Learning, Neural Networks, Speech Synthesis, Text-To-Speech (TTS), Grapheme-to-Phoneme (G2P), Expressive Speech, Pronunciation Variation, French, Sequence-to-Sequence, LSTM, Phoneme Error Rate, Attention Mechanism, OpenNMT, Emotional Expressivity, Mute "e".

Frequently Asked Questions

What is the primary research goal of this work?

The primary goal is to investigate how pronunciation variations (segmental level) contribute to the perception of emotions in French speech synthesis and whether deep neural networks can model these variations effectively to enhance expressivity.

What are the central thematic fields?

The study centers on Text-To-Speech synthesis, Grapheme-to-Phoneme conversion, deep learning, sequence-to-sequence modeling, and the phonetic analysis of emotional and expressive speech in French.

Which scientific method is used to develop the models?

The research uses an encoder-decoder sequence-to-sequence (Seq2Seq) architecture based on Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) cells, implemented within the OpenNMT toolkit.

What does the main part of the report cover?

The main part documents extensive experiments, starting with whole-sentence level training and progressing to optimized short-segment models, incorporating word boundary information and pre-training techniques to minimize the Phoneme Error Rate.

What do the most significant keywords describe?

The keywords highlight the intersection of advanced deep learning techniques (like RNNs/LSTMs) and linguistic specificities, specifically focusing on expressive speech synthesis and G2P conversion for the French language.

What is the overall conclusion regarding the research?

The work demonstrates that using short phoneme segments and pre-trained models leads to significant improvements in predicting expressive pronunciation, thereby validating the proposed architecture for expressive speech synthesis.

How does the usage of short segments impact model performance?

The experiments showed that splitting sequences at interword pauses into shorter segments significantly reduces the Phoneme Error Rate (PER) compared to training on long, whole-sentence sequences, as it makes the learning process more manageable for the model.

What role does the "Caroline" corpus play?

The Caroline expressive speech corpus provides the necessary labeled data (neutral and 6 emotional expressions) for the French language, enabling the actress-based "exercise in style" data to be used for training the G2P conversion models.