Development of a Robust Sign Language Translation System Using an Ensemble of Efficient Net Models for Enhanced Recognition Accuracy and Real-Time Performance

Sign language serves as the primary communication medium for over 466 million people with disabling hearing loss worldwide, projected to exceed 900 million by 2050 [1]. As a complete linguistic system with unique grammar and cultural nuances, it poses significant challenges for automated translation [2]. Deep learning has advanced SLT from isolated gesture recognition using convolutional neural networks (CNNs) [3] to sequence modeling with recurrent networks and transformers [4], [5]. Ensemble approaches leverage architectural diversity for improved performance [6], yet challenges persist in balancing accuracy, real-time efficiency, continuous signing handling, dataset diversity, and multilingual generalization [2], [7]. This study addresses these gaps through a hybrid ensemble framework. Objectives include developing an efficient architecture, enhancing continuous interpretation, improving robustness via augmentation, validating performance, and assessing deployability. Contributions comprise a novel adaptive fusion ensemble, superior continuous recognition (F1=0.82), benchmark-setting metrics (98.6% accuracy, 65 ms latency), and deployment guidelines. The work holds broad impact for education, healthcare, employment, and social inclusion [8], while preserving linguistic diversity and informing multimodal AI advancements [9].

Related Work

SLT research has progressed from hand-crafted features and hidden Markov models to deep learning paradigms [2]. CNNs like Efficient Net [10] and ResNet [11] dominate spatial feature extraction [3], [6]. Temporal modeling employs LSTMs [12], TCNs, or transformers [4], [5]. Hybrid CNN-transformer models integrate strengths [13], while ensembles enhance robustness [9], [6]. Multimodal inputs, including MediaPipe key points [14] and sensor fusion [15], improve invariance. Data augmentation mitigates scarcity [9], with metrics encompassing accuracy, BLEU [16], WER, F1, and latency [2]. Persistent gaps include real-time constraints, continuous signing, limited diversity, cross-linguistic transfer, and deployment robustness [7], [2]. This work bridges them via a diverse ensemble with key point preprocessing and comprehensive augmentation.

METHODOLOGY

System Architecture

The ensemble integrates:

EfficientNetB0: Lightweight spatial extractor with MBConv blocks; ImageNet-pretrained [10]. ResNet50: Deep residual network for hierarchical features; ImageNet-pretrained [11]. Transformer: 6-layer encoder-decoder (8 heads, 512-dim) with positional encoding and cross-attention for sequences [4]. Attention Fusion Layer: Dynamically weights features based on confidence and context [9]. Inputs are key point sequences and frames; outputs are gloss predictions or text translations.

Datasets

Combined WLASL [17] and MSASL [6]: 8,500 videos, 2,500 glosses, 150+ signers, 45 hours. Stratified split: 60% train, 20% validation, 20% test. Preprocessing and Augmentation Key points: Media Pipe Holistic extracts 42 hand, 468 face, 33 pose landmarks [14]. Normalization: 224×224 RGB frames. Segmentation: Motion thresholding for continuous sequences. Augmentation: Spatial (rotation ±15°, scaling 0.85-1.15×), temporal (speed 0.8-1.2×), occlusion, noise, blur. 3.4 Training and Implementation NVIDIA RTX 3090; TensorFlow 2.10. Adam optimizer (lr=10^{-4}-10^{-3}); batch size 32; 70 epochs with early stopping. Staged training: isolated components, joint fine-tuning, ensemble fusion. Combined loss: cross-entropy (classification/sequence) + attention regularization.

RESULTS

Training converged at epoch 67 (68 hours). Validation/test accuracy: 98.6%. Comparison with State-of-the-Art Table 1 compares key methods.

Table 1. Performance comparison.

Superior in BLEU (+10.3% over [6]), F1, and latency (-46% vs. [6]).

4.2 Ablation Study Table 2 shows component impacts.

Table 2. Ablation results.