[논문]딥러닝 방식의 웨어러블 센서를 사용한 미국식 수화 인식 시스템

정택위; 김범준

doi:10.13067/jkiecs.2020.15.2.291

딥러닝 방식의 웨어러블 센서를 사용한 미국식 수화 인식 시스템
American Sign Language Recognition System Using Wearable Sensors with Deep Learning Approach 원문보기

한국전자통신학회 논문지 = The Journal of the Korea Institute of Electronic Communication Sciences, v.15 no.2, 2020년, pp.291 - 298

정택위 (계명대학교 전자전기공학과) , 김범준 (계명대학교 전자전기공학과)

초록
AI-Helper

수화는 청각 장애인이 다른 사람들과 의사소통할 수 있도록 설계된 것이다. 그러나 수화는 충분히 대중화되어 있지 않기 때문에 청각 장애인이 수화를 통해서 일반 사람들과 원활하게 의사소통하는 것은 쉽지 않은 문제이다. 이러한 문제점에 착안하여 본 논문에서는 웨어러블 컴퓨팅 및 딥러닝 기반 미국식 수화인식 시스템을 설계하고 구현하였다. 이를 위해서 본 연구에서는 손등과 손가락에 장착되는 총 6개의 IMUs(Inertial Measurement Unit) 센서로 구성된 시스템을 구현하고 이를 이용한 실험을 수행하여 156개 특징이 수집된 데이터 추출을 통해서 총 28개 단어에 대한 미국식 수화 인식 방법을 제안하였다. 특히 LSTM (Long Short-Term Memory) 알고리즘을 사용하여 최대 99.89%의 정확도를 달성할 수 있었고 향후 청각 장애인들의 의사소통에 큰 도움이 될 것으로 예상된다.

Abstract ▼ AI-Helper

Sign language was designed for the deaf and dumb people to allow them to communicate with others and connect to the society. However, sign language is uncommon to the rest of the society. The unresolved communication barrier had eventually isolated deaf and dumb people from the society. Hence, this study focused on design and implementation of a wearable sign language interpreter. 6 inertial measurement unit (IMU) were placed on back of hand palm and each fingertips to capture hand and finger movements and orientations. Total of 28 proposed word-based American Sign Language were collected during the experiment, while 156 features were extracted from the collected data for classification. With the used of the long short-term memory (LSTM) algorithm, this system achieved up to 99.89% of accuracy. The high accuracy system performance indicated that this proposed system has a great potential to serve the deaf and dumb communities and resolve the communication gap.

주제어

표/그림 (5)

그림 Fig. 1 Proposed smart wearable sensors
표 Table 1. Proposed list of ASL words
표 Table 2. List of sensor data categories
그림 Fig. 2 Proposed LSTM model
그림 Fig. 3 Experiments results

AI 본문요약
AI-Helper

* AI 자동 식별 결과로 적합하지 않은 문장이 있을 수 있으니, 이용에 유의하시기 바랍니다.

제안 방법

In this study, 4 types of raw data were collected from the sensors during the experiments, namely accelerometer (A), gyroscope (G), orientation (O), and quaternion (Q). Accelerometer and gyroscope data were acquired from the IMU BNO055 sensors directly, however, orientation and quaternion data were calculated by utilizing sensor fusion computation. Then all these data were further reorganized and grouped into 15 sensor data categories by adopting leave-one-out techniques.
In this study, 4 types of raw data were collected from the sensors during the experiments, namely accelerometer (A), gyroscope (G), orientation (O), and quaternion (Q). Accelerometer and gyroscope data were acquired from the IMU BNO055 sensors directly, however, orientation and quaternion data were calculated by utilizing sensor fusion computation.
standard deviation (STD) and mean (MEAN). STD was measured to observe degree of movement, and MEAN was calculated to observe the average or arithmetic mean value of one window size. Then, all these features were further normalized between a range of 0 to 1 in order to reduce the processing time and data complexity.
The objective of this study targeted to design and implement a system with smart wearable sensors that able to recognized dynamic word-based ASL signs which involved movements. The system aimed to serve the deaf and dumb communities as a sign language interpreter that able decode sign language into text or audio to mass public.
Accelerometer and gyroscope data were acquired from the IMU BNO055 sensors directly, however, orientation and quaternion data were calculated by utilizing sensor fusion computation. Then all these data were further reorganized and grouped into 15 sensor data categories by adopting leave-one-out techniques.Table 2 showed the list of sensor data categories.
This study proposed the state-of-art time series data classification model, long short-term memory (LSTM). In comparison of typical recurrent neural network, LSTM introduced functional gates to overcome the vanishing and exploding gradient for a long sequence, namely forget gate (f), input gate (i), and output gate (o).

대상 데이터

There were 12 subjects performed data collection in this experiment. The experiments were under supervision of researcher(s) in order to make the data collection was obtained correctly and safely.

이론/모형

The 27 word-based ASL gestures were showed as Table 1, all of these gestures were selected based on the ASL components by William Stokoe’s terminology.

성능/효과

Conclusively, the experiment results indicated that integration of four sensor data had the best system performance. However, among all proposed sensor data categories, gyroscope was found had the least modality to represent hand and finger movement.
95% only. On the hand, standard deviation (STD) showed better performance than mean (MEAN) feature, 99.23% and 98.88% respectively. While, the combination of both features had even higher mean accuracy of 99.
The experiment results indicated that the integrated of four sensors data had the highest mean accuracy of 99.89% with STD and MEAN features.
[13] added two pressure sensors which were mounted on the top and the left of middle fingertip. The proposed study recognized 26 ASL fingerspelling letters using support vector machine (SVM) classifier had achieved the highest accuracy of 98.2%. Other than flex sensors, Mummadi et al.
The study revealed the highest accuracy rate of 94.1% by including all features extracted from three cameras, while the features that extracted from color camera delivered the lowest accuracy of 60.1% only.
[8] adopted a stereo camera for hand gesture recognition. This study achieved 98.6% of accuracy for 10 Arabic numbers recognition by skin segmented technique using Gaussian Mixture Model (GMM). The technique successfully overcame the difficulties of handling occlusion and overlapping regions between hands and faces.
[7] proposed a low cost marker-less hand gesture recognition system using webcam. This study utilized contours, convexity defects and Haar cascade to performance hand detection, and reported that complex background was an impactful factor to cause low detection accuracy. Then, Elmezain et al.
89% with STD and MEAN features. While the single sensor data categories, Q, O, G, and A had similar lowest accuracy compare to the rest of sensor data categories, especially the G data with MEAN features, 96.95% only. On the hand, standard deviation (STD) showed better performance than mean (MEAN) feature, 99.

참고문헌 (17)

T. Chong and B. Lee, "American Sign Language Recognition Using Leap Motion Controller with Machine Learning Approach," Sensors, vol. 18, no. 10, 2018, pp. 35-54.
M. J. Cheok, "A review of hand gesture and sign language recognition techniques," Int. J. Mach. Learn. Cybern., vol. 10, issue 1, 2019, pp. 131-153.

상세보기
M. A. Ahmed, B. B. Zaidan, and A. A. Zaidan, "A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017," Sensors, vol. 18, no. 7, 2018, pp. 1-44.
I. Infantino, R. Rizzo, and S. Gaglio, "A Framework for Sign Language Sentence Recognition by Commonsense Context," IEEE Trans. Syst. Man, Cybern. vol. 37, no. 5, 2007, pp. 1034-1039.
L. Ding and A. M. Martinez, "Modelling and Recognition of the Linguistic Components in American Sign Language Title," Image Vis. Comput., vol. 27, no. 12, 2009, pp. 1826-1844.

상세보기
H. Lane and F. Grosjean, Recent perspective on American Sign Language. New York, Pyschology Press, 2017.
A. Haria, A. Subramanian, N. Asokkumar, and S. Poddar, "ScienceDirect ScienceDirect Hand Gesture Recognition for Human Computer Interaction," Procedia Comput. Sci., vol. 115, 2017, pp. 367-374.
M. Elmezain and A. Al-hamadi, "A Hidden Markov Model-Based Isolated and Meaningful Hand Gesture Recognition," Int. J. Electr. Comput. Syst. Eng., vol. 3, no. 3, 2009, pp. 156-163.
P. Molchanov, S. Gupta, K. Kim, and K. Pulli, "Multi-sensor System for Driver's Hand-Gesture Recognition," In Proc. 2015 11th IEEE Int. Conf. Work. Autom. Face Gesture Recognit., vol. 1, 2015, pp. 1-8.
C. Preetham, G. Ramakrishnan, S. Kumar, and A. Tamse, "Hand Talk- Implementation of a Gesture Recognizing Glove," In Proc. 2013 Texas Instruments India Educators' Conference, Bangalore, India, 2013, pp. 328-331.
K. Patil, G. Pendharkar, and P. G. N. Gaikwad, "American Sign Language Detection," Int. J. Sci. Res. Publ., vol. 4, no. 11, 2014, pp. 4-9.
X. Wang, M. Xia, H. Cai, Y. Gao, and C. Cattani, "Hidden-Markov-Models-Based Dynamic Hand Gesture Recognition," Math. Probl. Eng., vol. 2012, 2012, p. 11.
B. G. Lee and S. M. Lee, "Smart Wearable Hand Device for Sign Language Interpretation System With Sensors Fusion," IEEE Sens. J., vol. 18, no. 3, 2018, pp. 1224-1232.

상세보기
C. K. Mummadi, F. Leo, K. Verma, S. Kasireddy, P. Scholl, J. Kempfle, and K. Laehoven, "Real-Time and Embedded Detection of Hand Gestures with an IMU-Based Glove," Informatics, vol. 5, no. 2, 2018, pp. 1-18.
J. Kim and W. Lee, "An User-aware System using Visible Light Communication," J. of the Korea Institute of Electronic Communication Sciences, vol. 14, no. 4, 2019, pp. 715-722.

원문보기 상세보기
J. Jo, "Effectiveness of Normalization Pre-Processing of Big Data to the Machine Learning Performance," J. of the Korea Institute of Electronic Communication Sciences, vol. 14, no. 3, 2019, pp. 547-552.

원문보기 상세보기
J. Jo, "Performance Comparison Analysis of AI Supervised Learning Methods of Tensorflow and Scikit-Learn in the Writing Digit Data," J. of the Korea Institute of Electronic Communication Sciences, vol. 14, no. 4, 2019, pp. 701-706.

원문보기 상세보기

표제어: PCR

동의어: Packet Collision Rate

용어 설명 출처 목록 (6)

용어 설명: PCR은 세균 특이성이 있는 primer를 이용하여 적은 수의 세균이 있을지라도 쉽게 검출할 수 있는 유용한 방법이며, 이를 이용하여 구강 내 치면세균막이나 타액에서 직접 세균을 검출할 수 있게 되었다[8].

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증