Audio-visual speech recognition using depth information from the Kinect in noisy video conditions
Ημερομηνία
2012Λέξη-κλειδί
Επιτομή
In this paper we build on our recent work, where we successfully incorporated facial depth data of a speaker captured by the Microsoft Kinect device, as a third data stream in an audio-visual automatic speech recognizer. In particular, we focus our interest on whether the depth stream provides sufficient speech information that can improve system robustness to noisy audio-visual conditions, thus studying system operation beyond the traditional scenarios, where noise is applied to the audio signal alone. For this purpose, we consider four realistic visual modality degradations at various noise levels, and we conduct small-vocabulary recognition experiments on an appropriate, previously collected, audiovisual database. Our results demonstrate improved system performance due to the depth modality, as well as considerable accuracy increase, when using both the visual and depth modalities over audio only speech recognition.
Collections
Related items
Showing items related by title, author, creator and subject.
-
Scattering vs. Discrete Cosine Transform Features in Visual Speech Processing
Marcheret E., Potamianos G., Vopicka J., Goel V. (2015)Appearance-based feature extraction constitutes the dominant approach for visual speech representation in a variety of problems, such as automatic speechreading, visual speech detection, and others. To obtain the necessary ... -
Resource-efficient TDNN Architectures for Audio-visual Speech Recognition
Koumparoulis A., Potamianos G., Thomas S., da Silva Morais E. (2021)In this paper, we consider the problem of resource-efficient architectures for audio-visual automatic speech recognition (AVSR). Specifically, we complement our earlier work that introduced efficient convolutional neural ... -
Deep View2View Mapping for View-Invariant Lipreading
Koumparoulis A., Potamianos G. (2019)Recently, visual-only and audio-visual speech recognition have made significant progress thanks to deep-learning based, trainable visual front-ends (VFEs), with most research focusing on frontal or near-frontal face videos. ...