Fingerspelled alphabet sign recognition in upper-body videos

Abstract

Fingerspelling is a crucial part of sign-based communication, however its recognition remains a challenging and mostly overlooked computer vision problem. To address it, this paper presents a system that recognizes the 24 static fingerspelled alphabet signs of the American Sign Language. The system consists of two algorithmic stages, comprising an efficient preprocessing phase that generates candidate hand-region proposals, followed by their deep-learning based classification. Specifically, the first stage exploits own earlier work on hand detection and segmentation in videos that also contain the signer's face, allowing face detection to drive skin-tone based hand segmentation, with motion further utilized to localize hands, extending it with a peak detection module that yields proposal regions likely to contain the signs of interest. These regions are then classified by a variant of a convolutional neural network that extends traditional convolutions to quadratic operations on the inputs, being, to our knowledge, the first application of such architecture to this task. Both system stages are evaluated on three well-known fingerspelling corpora, significantly outperforming a number of alternative approaches under both multi-signer and signer-independent experimental frameworks. © 2019 IEEE