CVPR 2011 workshop on gesture recognition

Segmentation-robust representations, matching, modeling for sign
language recognition
Sudeep Sarkar
University of South Florida

Distinguishing true signs from transitional, extraneous movements made by the signer as s/he moves from one sign to the next is a serious hurdle in the design of continuous Sign Language recognition systems. This problem is further compounded by the ambiguity of segmentation and occlusions, resulting in propagation of errors to higher levels. This talk will describe our experience with representations and matching methods, particularly those that can handle errors in low-level segmentation methods and uncertainties in segmentation of signs in sentences.  We have formulated a novel framework that can address both these problems (i) using a nested level-building-based dynamic programming approach, when there is dearth of training data, and (ii) using a HMM-based approach generalized to handle multiple possible observations, when we have statistical models of signs. We will also discuss an automated approach to both extract and learn models for continuous signs from continuous sentences in a weakly unsupervised manner. This can help build training data for the recognition process. Our publications that discuss these issues can be found at