Jennifer Santoso, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm), Shoji Makino(Waseda Univ./Univ. of Tsukuba)

Speech emotion recognition (SER) has been actively studied in the recent decade and has achieved promising results. Most state-of-the-art SER methods are based on a classification approach that ultimately outputs the softmax probability over different emotion classes. On the other hand, we have recently introduced an anomalous sound detection approach to improve the SER performance of the neutral class. It uses a neutral speech detector consisting of an autoencoder that reconstructs acoustic and text features in latent space and is trained using only neutral speech data. The experimental result confirmed that the reconstruction error could be successfully used as a cue to decide whether or not the class is neutral and suggested that it could be applied to other emotion classes. In this paper, we propose an SER method based on the reconstruction of acoustic and text features in latent space, in which the reconstructor for different emotion classes, including the neutral class, is used. The proposed method selects the emotion class with the lowest normalized reconstruction error as the SER result. Unlike the classifier approach, one reconstructor is dedicated to each emotion class and trained using only the data of the target emotion class. Therefore, the reconstructor can be trained without being affected by imbalanced training data and also facilitates the application of data augmentation to only a specific emotion class. Our experimental result obtained using the IEMOCAP dataset showed that the proposed method improved the class-average weighted accuracy by 1.7% to 77.8% compared with the stateof-the-art SER methods.

Jennifer Santoso, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm), Shoji Makino(Waseda Univ./Univ. of Tsukuba)

Speech emotion recognition (SER) is essential for understanding a speaker’s intention. Recently, some groups have attempted to improve SER performance using a bidirectional long short-term memory (BLSTM) to extract features from speech sequences and a self-attention mechanism to focus on the important parts of the speech sequences. SER also benefits from combining the information in speech with text, which can be accomplished automatically using an automatic speech recognizer (ASR), further improving its performance. However, ASR performance deteriorates in the presence of emotion in speech. Although there is a method to improve ASR performance in the presence of emotional speech, it requires the fine-tuning of ASR, which has a high computational cost and leads to the loss of cues important for determining the presence of emotion in speech segments, which can be helpful in SER. To solve these problems, we propose a BLSTM-and-self-attention-based SER method using self-attention weight correction (SAWC) with confidence measures. This method is applied to acoustic and text feature extractors in SER to adjust the importance weights of speech segments and words with a high possibility of ASR error. Our proposed SAWC reduces the importance of words with speech recognition error in the text feature while emphasizing the importance of speech segments containing these words in acoustic features. Our experimental results on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset reveal that our proposed method achieves a weighted average accuracy of 76.6%, outperforming other state-of-the-art methods. Furthermore, we investigated the behavior of our proposed SAWC in each of the feature extractors.

Jennifer Santoso, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm), Shoji Makino(Waseda Univ./Univ. of Tsukuba)

In recent years, classification-based speech emotion recognition (SER) methods have achieved high overall performance. However, these methods tend to have lower performance for neutral speeches, which account for a large proportion in most practical situations. To solve the problem and improve the SER performance, we propose a neutral speech detector (NSD) based on the anomaly detection approach, which uses an autoencoder, the intermediate layer output of a pretrained SER classifier and only neutral data for training. The intermediate layer output of a pretrained SER classifier enables the reconstruction of both acoustic and text features, which are optimized for SER tasks. We then propose the combination of the SER classifier and the NSD used as a screening mechanism for correcting the class probability of the incorrectly recognized neutral speeches. Results of our experiment using the IEMOCAP dataset indicate that the NSD can reconstruct both the acoustic and textual features, achieving a satisfactory performance for use as a reliable screening method. Furthermore, we evaluated the performance of our proposed screening mechanism, and our experiments show significant improvement of 12.9% in the F-score of the neutral class to 80.3%, and 8.4% in the class-average weighted accuracy to 84.5% compared with state-of-the-art SER classifiers.

Jumon Nozaki, Tatsuya Kawahara(Kyoto Univ.), Kenkichi Ishizuka, Taiichi Hashimoto(Revcomm)

Conventional automatic speech recognition systems do not produce punctuation marks which are important for the readability of the speech recognition results. They are also needed for subsequent natural language processing tasks such as machine translation. There have been a lot of works on punctuation prediction models that insert punctuation marks into speech recognition results as post-processing. However, these studies do not utilize acoustic information for punctuation prediction and are directly affected by speech recognition errors. In this study, we propose an end-to-end model that takes speech as input and outputs punctuated texts. This model is expected to predict punctuation robustly against speech recognition errors while using acoustic information. We also propose to incorporate an auxiliary loss to train the model using the output of the intermediate layer and unpunctuated texts. Through experiments, we compare the performance of the proposed model to that of a cascaded system. The proposed model achieves higher punctuation prediction accuracy than the cascaded system without sacrificing the speech recognition error rate. It is also demonstrated that the multi-task learning using the intermediate output against the unpunctuated text is effective. Moreover, the proposed model has only about 1/7th of the parameters compared to the cascaded system.

Heran Zhang, Masato Mimura, Tatsuya Kawahara(Kyoto Univ.), Kenkichi Ishizuka(RevComm)

In this paper, we investigate a speech emotion recognition based on multi-task learning (MTL) considering the style attribute. We also hypothesize the neutral expression, which has the largest number of samples, is not affected by the style, and thus propose a selective MTL method that applies MTL to emotion categories except for the neutral category. Experimental evaluations using the IEMOCAP database and a call center dataset confirm the effect of the combination of the two corpora, MTL, and the proposed selective MTL.

Santoso Jennifer、Yamada Takeshi(Univ. of Tsukuba)、Ishizuka Kenkichi、Hashimoto Taiichi(RevComm)、Makino Shoji(Waseda Univ./Univ. of Tsukuba).

In recent years, there have been several deep-learning-based methods to im- prove the performance of speech emotion recognition (SER). However, in most of these studies, the recognition of neutral speeches, which is the most common type of speech in practical settings, tend to have poor performance. One of the reasons is the wide data distribution for neutral speeches, which is harder to generalize than emotional speeches.
In several practical settings, such as business conversation analysis, most conversations do not contain emotions. Emotional speeches, therefore, are considered an unusual occurrence and might be an indicator of trouble or unanticipated events in the conversation. Therefore, by taking advantage of a large number of neutral speeches available, it is possible to tackle problems in the SER from the anomaly detection approach, where neutral speeches are considered normal and emotional speeches are considered anomalous.

加藤 集平、橋本 泰一 (RevComm)

電話などの音声通話においては、話し手がいくらクリーンな環境下で話していても、聞き手が雑音環境下にいると話し手の声を聞き取りづらくなることがある。このような状況において、音声強調によって（聞き手側での）音声了解度を向上させることは near-end listening enhancement (NELE) と呼ばれている。NELEの代表的な手法の一つにNELE-GANがあるが、その性能の検証は比較的少量かつ限られた話者の音声データを用いたものにとどまっている。本稿では、非常に大量かつ多様な音声データを用いた検証を行い、比較的少量のデータを用いる場合と比べて、音声了解度を同程度に保つかさらに向上させた上で、音声品質を大幅に向上させられることを明らかにした。

野崎樹文（京都大）、石塚賢吉、橋本泰一（RevComm）、河原達也（京都大）

一般に音声認識システムの出力するテキストには句読点が含まれていない。これは音声認識結果の可読性を下げる要因の一つである。また、機械翻訳や自動要約などの後続の自然言語処理タスクへの入力として望ましくない。本研究では、音声を入力として句読点の付いたテキストを End-to-End に認識するモデルを提案する。これにより音響情報を使用しながら、音声認識誤りに対して頑健に句読点を予測することを目指す。評価実験は日本語と英語のデータセットを用いて行い、提案モデルの有効性を示す。さらに、BERTを用いて音声認識結果のテキスト情報のみから句読点を予測する従来のシステムとの性能を比較する。

Jennifer Santoso, Takeshi Yamada(Univ. of Tsukuba), Shoji Makino(Waseda Univ./Univ. of Tsukuba), Kenkichi Ishizuka, Takekatsu Hiramura(Revcomm)

Emotion recognition is essential for human behavior analysis and possible through various inputs such as speech and images. Considering the complexity of emotions, speech emotion recognition(SER) is a challenging task. Recently, automatic speech recognition (ASR) has played a role in obtaining text information from speech. The combination of speech and ASR results has improved the SER performance. However, ASR results are highly affected by speech recogni- tion errors. Although there is a method to improve ASR perfor- mance on emotional speech, it requires the fine-tuning of ASR, which is costly. To mitigate the errors in SER using ASR sys- tems, we propose the use of the combination of a self-attention mechanism and a word-level confidence measure (CM), which indicates the reliability of ASR results, to reduce the impor- tance of words with a high chance of error. Experimental results confirmed that the combination of self-attention mechanism and CM reduced the effects of incorrectly recognized words in ASR results, providing a better focus on words that determine emo- tion recognition. Our proposed method outperformed the state- of-the-art methods on the IEMOCAP dataset.

平村 健勝(RevComm)、篠崎 隆宏(東工大)