Abstract

People with speech and hearing impairments find it hard to communicate with others. The Sign Language Gesture Recognition and Translation system in this paper is meant to help people with speech impairments interact easily with others. The system can pick out five important sign language gestures: Yes, No, I Love You, Thank You, Hello, and OK, and convert them to clear text messages right away. The main part of the system uses a Long Short-Term Memory (LSTM) neural network, Python, OpenCV, and Mediapipe to ensure both accurate and detailed hand tracking. Using computer vision, the program effectively reads each video frame from the webcam and detects the movements of hands very accurately and quickly. The fact that the interface just needs hand movements in front of any webcam means that this technology is easy to use and not expensive. By carrying out this research, we want to help the speech and hearing impaired communicate better, through easy and effective solutions. Since the results were encouraging, the system can be used in real-life situations and applied to a bigger range of sign vocabulary.

References

• S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
• A. Graves, S. Fern´andez, and J. Schmidhuber, “Bidirectional LSTM networks for improved phoneme classification,” in Proc. Int. Conf. Artificial Neural Networks (ICANN), 2005, pp. 799–804.
• Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
• F. Chollet, Deep Learning with Python, 2nd ed., Manning Publications, 2021.
• G. Khartheesvar, M. Kumar, A. K. Yadav, and D. Yadav, “Automatic Indian Sign Language Recognition using MediaPipe Holistic and LSTM Network,” Multimedia Tools and Applications, vol. 83, no. 20, pp. 58329–58348, 2023.
• P. Swetha and K. Sucharitha, “Sign Language Recognition Utilizing LSTM and MediaPipe for Dynamic Gestures of ISL,” Int. J. Res. Sci. Eng. Technol., vol. 10, no. 6, 2023.
• K. Swathi, V. Reddy, and P. Divya, “ISL Sign Language Recognition Using LSTM-Driven Deep Learning Model,” J. Electr. Syst., vol. 20, no. 3, 2024.
• P. Tripathi, A. Shukla, and A. S. Negi, “Gesture-to-Text Translation Using SURF for Indian Sign Language,” Appl. Syst. Innov., vol. 6, no. 2, 2023.
• P. Navendu and V. Sahula, “Word Level Sign Language Recognition using MediaPipe and LSTM-GRU Network,” TechRxiv, Jul. 2024.
• D. KhajehPourian and A. D. Zarei, “Sign language recognition using modified deep learning network: CNNSa-LSTM,” Scientific Reports, vol. 14, 2024.
• D. Patel and S. Patel, “Dynamic Indian Sign Language Recognition Based on Enhanced LSTM with Attention Mechanism,” SSR-IJECE, vol. 11, no. 2, Feb. 2024.
• Y. Zhang and X. Jiang, “Recent Advances on Deep Learning for Sign Language Recognition,” Comput. Model. Eng. Sci., vol. 170, 2024.
• H. Hu et al., “SignBERT: Pre-Training of Hand-Model- Aware Representation for Sign Language Recognition,” arXiv preprint arXiv:2110.05382, 2021.
• K. Shenoy, T. Dastane, V. Rao, and D. Vyavaharkar, “Real-time Indian Sign Language (ISL) Recognition,” arXiv preprint arXiv:2108.10970, 2021.