AI-Driven Indian Sign Language Recognition Using Hybrid CNN-BiLSTM Architecture for Divyangjan
Article Sidebar
Main Article Content
Abstract
Barriers in expressive and receptive communication for individuals with hearing and speech impairments remain a significant challenge to achieving equitable social interaction and digital accessibility. These limitations restrict their participation in everyday conversations, education, and professional environments, emphasizing the urgent need for intelligent assistive communication technologies. This research introduces an AI-driven real-time Indian Sign Language (ISL) recognition framework that integrates advanced computer vision and deep learning techniques to translate hand gestures into textual outputs. The primary objective of this study is to design a lightweight, accurate, and real-time sign language translator suitable for Divyangjan, ensuring inclusivity in educational and social interactions. The proposed research employs Mediapipe-based hand landmark extraction to detect 21 key points in each gesture frame, followed by preprocessing and normalisation to create robust spatial representations. A hybrid Convolutional Neural Network processes these landmark vectors through a Bidirectional Long Short-Term Memory (CNN–BiLSTM) model that captures both spatial and temporal dependencies in gesture motion, allowing it to recognise static and dynamic gestures such as “J” and “Z.” The system was trained on a self-collected dataset of over 26,000 gesture images covering all 26 ISL alphabets. Experimental analysis demonstrates that the proposed model achieved 97.8% accuracy, 97.4% precision, 97.2% recall, 97.3% F1-score, and a validation loss of only 0.08, outperforming traditional classifiers such as Random Forest and SVM by a significant margin. The trained model performs robustly under varying lighting, background, and hand orientation conditions, ensuring high reliability for real-world deployment. The novelty of this study lies in the fusion of Mediapipe landmark extraction with a temporal deep learning framework for continuous ISL gesture translation within an interactive CustomTkinter GUI. This human-centric, computationally efficient design enables accessible, real-time communication for the deaf and hard-of-hearing community, contributing to socially assistive AI systems that promote digital inclusivity and empowerment for Divyangjan.
Downloads
Article Details
Section
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
How to Cite
References
Aly S and Aly W, (2020), DeepArSLR: Signer-Independent Deep Learning for Arabic Sign Language Recognition in the Wild, IEEE Access, 8, 83199–83212. Available from: https://doi.org/10.1109/ACCESS.2020.2990699
Bragg D, Tabrizi M, Toda A, Vogler C, Kacorri H (2020). Exploring Collection of Sign Language Datasets Through Crowdsourcing, Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). Available From: https://doi.org/10.1145/3373625.3417024
Ma Y, Song J, Ji M, (2022), Two-Stream Mixed Convolutional Neural Network for Sign Language Recognition, Sensors, 22(16), 5959. Available from: https://doi.org/10.3390/s22165959
Kothadiya D, Bhatt C, Sapariya K, Patel K, Gil-González A-B, Corchado JM, (2022), Deepsign: Sign Language Detection and Recognition Using Deep Learning, Electronics, 11(11), 1780. Available From: https://doi.org/10.3390/electronics11111780
Srinivas K, Khan AA, Alavandar S, Nandhini K, (2022), Deep Learning-Based System for Real-Time Indian Sign Language Recognition, Wireless Personal Communications, 126(2), 1385–1413. Available From: https://doi.org/10.1007/s11277-021-09152-1
Aloysius N, Geetha M, Nedungadi P, (2021), Incorporating Relative Position Information in Transformer-Based Sign Language Recognition and Translation, IEEE Access, 9, 145929–145942. Available from: https://doi.org/10.1109/ACCESS.2021.3122921
Yoo S, Park J, Yang HJ, Kang H, (2023), A Korean Sign Language Recognition Model Using OpenPose and Transformers, Applied Sciences, 13(5), 3029. Available From: https://doi.org/10.3390/app13053029
Morales A, Akula A, Yim D, Park D, Lee S, (2023), Segmentation of Hand Regions for Gesture Recognition Using MediaPipe, Virtual Reality, 27, 3179–3195. Available from: https://doi.org/10.1007/s10055-023-00858-0
Yin P, Wang J, Zhang W, (2025), Deep Learning Classification of Human Hand Landmarks Using MediaPipe Hand-Landmarks Model, Electronics, 14(4), 704. Available From: https://doi.org/10.3390/electronics14040704
Kumari D, Alnaim RS (2024). Isolated Video-Based Sign Language Recognition Using a Hybrid CNN-LSTM Framework Based on an Attention Mechanism. Electronics, 13(7), 1229. Available from: https://doi.org/10.3390/electronics13071229
Abdul W, Alsulaiman M, Amin SU, Faisal M, Ghaleb H, et al. (2021). Intelligent Real-Time Arabic Sign Language Classification Using Attention-Based Inception and BiLSTM, Computers & Electrical Engineering, 95, 107395. Available From: ttps://doi.org/10.1016/j.compeleceng.2021.107395
Noor TH, Al-Nuaimy W, Al-Ataby A, (2024), Real-Time Arabic Sign Language Recognition Using a Hybrid CNN–LSTM Approach, Sensors, 24(11), 3683. Available from: https://doi.org/10.3390/s24113683
Rastgoo R, Kiani K, Escalera S, (2020), Hand Sign Language Recognition Using Multi-View Hand Skeleton, Expert Systems with Applications, 150, 113336. Available from: https://doi.org/10.1016/j.eswa.2020.113336
Jiang S, Sun B, Wang L, Bai Y, Li K, Fu Y, (2021), Skeleton Aware Multi-Modal Sign Language Recognition, CVPR Workshops. Available From: https://doi.org/10.1109/CVPRW53098.2021.00380
Li D, Rodriguez-Opazo C, Yu X, Li H, (2020), Word-Level Deep Sign Language Recognition from Video: A New Large-Scale Dataset and Methods Comparison, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 1459–1469. Available From: https://doi.org/10.1109/WACV45572.2020.9093512
Wali A, Belguith S, Bouhlel MS, (2023), Recent Progress in Sign Language Recognition: A Review, Machine Vision and Applications, 34, 82. Available from: https://doi.org/10.1007/s00138-023-01479-y
Tan S, Zhao D, Wong KM, (2024), A Review of Deep Learning-Based Approaches to Sign Language Recognition, Advanced Robotics, 38(17–18), 1091–1117. Available From: https://doi.org/10.1080/01691864.2024.2442721
Venugopalan A, Srivastava S, Karthik R, (2023), Applying Hybrid Deep Neural Network for the Recognition of Medical Emergency Signs in Indian Sign Language, Arabian Journal for Science and Engineering, 48, 11059–11071. Available from: https://doi.org/10.1007/s13369-022-06843-0
Hu H, Zhao W, Zhou W, Li H, (2023), SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language Understanding, arXiv. Available From: https://doi.org/10.48550/arXiv.2305.04868
Sandoval-Castañeda M, et al. (2023). Self-Supervised Video Transformers for Isolated Sign Language Recognition, arXiv. Available From: https://doi.org/10.48550/arXiv.2309.02450
Al-Qurishi M, Baz A, Alhakami H, Muthanna A, Khan J, et al. (2021). Sign Language Recognition Systems for the Deaf and Dumb: A Review, IEEE Access, 9, 126917–126951. Available From: https://doi.org/10.1109/ACCESS.2021.3110912
Alsharif B, Alalwany E, Ibrahim A, Mahgoub I, Ilyas M, (2025), Real-Time American Sign Language Interpretation Using Deep Learning and Keypoint Tracking, Sensors, 25(7), 2138. Available from: https://doi.org/10.3390/s25072138
Zhang Y, Wang Y, Li F, (2024), Sign Language Recognition Based on CNN-BiLSTM Using RF Signals, IEEE Access. Available From: https://doi.org/10.1109/ACCESS.2024.3517417
Papastratis I, Chatzikonstantinou C, Konstantinidis D, Dimitropoulos K, Daras P, (2021), Artificial Intelligence Technologies for Sign Language, Sensors, 21(17), 5843. Available from: https://doi.org/10.3390/s21175843
Zhang L, (2025), Recognising American Sign Language Gestures Efficiently with Transformer-Based Self-Attention, Scientific Reports, 15, 6344. Available From: https://doi.org/10.1038/s41598-025-06344-8
R. Arularasan, D. Balaji, S. Garugu, V. R. Jallepalli, S. Nithyanandh and G. Singaram, (2024), Enhancing Sign Language Recognition for Hearing-Impaired Individuals Using Deep Learning, 2024 International Conference on Data Science and Network Security (ICDSNS), Tiptur, India, pp. 1–6. Available From: https://doi.org/10.1109/ICDSNS62112.2024.10690989
Eldho K. J. and Nithyanandh S., (2024), Lung Cancer Detection and Severity Analysis with a 3D Deep Learning CNN Model Using CT-DICOM Clinical Dataset, Indian Journal of Science and Technology, 17(10), 899–910. Available From: https://doi.org/10.17485/IJST/v17i10.3085
Nithyanandh S., (2025), Object Detection and Analysis with Deep CNN and YOLOv8 in Soft Computing Frameworks, International Journal of Soft Computing and Engineering (IJSCE), 14(6), 19–27. Available From: https://doi.org/10.35940/ijsce.E3653.14060125
G. Indhumathi, P. S. Anil, A. Posiyya, H. R. Suresh and S. Navaneethan. Deep Learning-Based Tongue Biometrics for Secure Authentication in IoT-Driven Healthcare Systems, Smart & Sustainable Technology, 2025, 1-6. Available From: https://doi.org/10.1109/INCSST64791.2025.11210319
V. Nivedita, J. A. Joseph, D. T. Varghese, J. Sundaram and G. Ali. Multi-Modal Biometric Authentication Integrating Gait and Face Recognition for Mobile Security, Smart & Sustainable Technology, 2025, 1-6. https://doi.org/10.1109/INCSST64791.2025.11210361