[논문]지능형 전시 서비스 구현을 위한 멀티모달 감정 상태 추정 모형

이기천; 최소윤; 김재경; 안현철

doi:10.13088/jiis.2014.20.1.001

지능형 전시 서비스 구현을 위한 멀티모달 감정 상태 추정 모형
Multimodal Emotional State Estimation Model for Implementation of Intelligent Exhibition Services 원문보기

지능정보연구 = Journal of intelligence and information systems, v.20 no.1, 2014년, pp.1 - 14

이기천 (한양대학교 공과대학 산업공학과) , 최소윤 (국민대학교 비즈니스IT전문대학원) , 김재경 (경희대학교 경영대학) , 안현철 (국민대학교 경영대학 경영정보학부)

초록
AI-Helper

최근 관람객의 반응에 따라 실시간으로 대응하여 관객의 몰입과 만족도를 증대시키는 인터랙티브 전시 서비스에 대한 학계와 산업계의 관심이 높아지고 있다. 이러한 인터랙티브 전시 서비스를 효과적으로 구현하기 위해서는 관객의 반응을 통해 해당 관객이 느끼는 감정 상태를 추정할 수 있는 지능형 기술의 도입이 요구된다. 인간의 감정 상태를 추정하기 위한 시도들은 많은 연구들에서 이루어져 왔고, 그 중 대부분은 사람의 얼굴 표정이나 소리 반응을 통해 감정 상태를 추정하는 방식을 도입하고 있다. 하지만, 최근 소개되고 있는 연구들에 따르면 단일 반응이 아닌 여러 반응을 종합적으로 고려하는 이른바 멀티 모달(multimodal) 접근을 사용했을 경우, 인간의 감정 상태를 보다 정확하게 추정할 수 있다. 이러한 배경에서 본 연구는 키넥트 센서를 통해 측정되는 관객의 얼굴 표정, 몸짓, 움직임 등을 종합적으로 고려한 새로운 멀티모달 감정 상태 추정 모형을 제안하고 있다. 제안모형의 예측 기법으로는 방대한 양의 데이터를 효과적으로 처리하기 위해, 몬테칼로(Monte Carlo) 방법인 계층화 샘플링(stratified sampling) 방법에 기반한 다중회귀분석을 적용하였다. 제안 모형의 성능을 검증하기 위해, 15명의 피실험자로부터 274개의 독립 및 종속변수들로 구성된 602,599건의 관측 데이터를 수집하여 여기에 제안 모형을 적용해 보았다. 그 결과 10~15% 이내의 평균오차 범위 내에서 피실험자의 쾌/불쾌도(valence) 및 각성도(arousal) 상태를 정확하게 추정할 수 있음을 확인할 수 있었다. 이러한 본 연구의 제안 모형은 비교적 구현이 간단하면서도 안정성이 높아, 향후 지능형 전시 서비스 및 기타 원격학습이나 광고 분야 등에 효과적으로 활용될 수 있을 것으로 기대된다.

Abstract ▼ AI-Helper

Both researchers and practitioners are showing an increased interested in interactive exhibition services. Interactive exhibition services are designed to directly respond to visitor responses in real time, so as to fully engage visitors' interest and enhance their satisfaction. In order to install an effective interactive exhibition service, it is essential to adopt intelligent technologies that enable accurate estimation of a visitor's emotional state from responses to exhibited stimulus. Studies undertaken so far have attempted to estimate the human emotional state, most of them doing so by gauging either facial expressions or audio responses. However, the most recent research suggests that, a multimodal approach that uses people's multiple responses simultaneously may lead to better estimation. Given this context, we propose a new multimodal emotional state estimation model that uses various responses including facial expressions, gestures, and movements measured by the Microsoft Kinect Sensor. In order to effectively handle a large amount of sensory data, we propose to use stratified sampling-based MRA (multiple regression analysis) as our estimation method. To validate the usefulness of the proposed model, we collected 602,599 responses and emotional state data with 274 variables from 15 people. When we applied our model to the data set, we found that our model estimated the levels of valence and arousal in the 10~15% error range. Since our proposed model is simple and stable, we expect that it will be applied not only in intelligent exhibition services, but also in other areas such as e-learning and personalized advertising.

주제어

AI 본문요약
AI-Helper

* AI 자동 식별 결과로 적합하지 않은 문장이 있을 수 있으니, 이용에 유의하시기 바랍니다.

제안 방법

In addition, as far as we know, there have been no prior studies that use the Microsoft Kinect Sensor as a tool for collecting multimodal responses. Although this study proposes an emotional state estimation model for the purpose of interactive or intelligent exhibition services, it can also be applied to other domains such as e-learning and personalized advertisements. We expect that future studies in this field will use or develop on our proposed model.
Based on these promising results, we built a real interactive exhibition system (shown in [Figure 9]) that measures an individual’s sensibility by using the model proposed in this study.
Third, this study adopts a stratified sampling-based MRA as the prediction method. MRA is known to be a computationally simple and stable algorithm, so it was easy for us to convert our proposed model into an actual interactive exhibition system.
This study proposes a novel multimodal emotional state estimation model using human responses measured by the Microsoft Kinect Sensor. Our proposed model estimates time-variant levels of valence and arousal by using 272 input values, 121 facial points in 2D and 10 body points in 3D.
Using the Kinect Sensor, this study estimates peoples’ emotional states through their facial expressions and the movements of their head and shoulders.
We predict the true values of valence and arousal by using the averages of the regression vectors from the sampling-based regression. To illustrate the performance, we randomly choose fifty observations from the data set and predict their values of valence and arousal.

대상 데이터

Kinect Sensor is a motion sensing input device that was first introduced by Microsoft in 2010. It uses three kinds of sensors: an RGB camera, a depth sensor, and a multi-array microphone. Among these, the depth sensor, which consists of an infrared laser projector combined with a monochrome CMOS (Complementary Metal –Oxide–Semiconductor) sensor, can capture video data in 3D, a capability which enables motion or gesture tracking (Wikipedia, 2014).
The experimental system was developed using C#, and [Figure 2] demonstrates the control screen. Our experimental system is designed to track 121 facial points and 10 body points. Facial points are measured in 2D (x-y axes), while body points are measured in 3D (x-y-z axes).
To validate the effectiveness of the proposed model, we collected experimental data from 17 university and graduate school student volunteers from January 14 to 25, 2013. The subjects were exposed to a 23-minute video consisting of five short clips to stimulate various high or low states of valence or arousal.

데이터처리

The subjects were exposed to a 23-minute video consisting of five short clips to stimulate various high or low states of valence or arousal. As the subjects watched the video, their responses were measured and recorded by the Kinect Sensor. After watching the video, the subjects were asked to use the Feeltrace tool to annotate their emotional state(s) while they were watching the video.

참고문헌 (20)

Ahn, H., "GA-optimized Support Vector Regression for Better Emotional State Estimation Model," Proceedings of the 5th International Conference on Internet (ICONI 2013), (2013), 333-335.
Box, G. E. P. and. D. R. Cox, "An analysis of transformations," Journal of the Royal Statistical Society, Series B, Vol. 26, No. 2(1964), 211-252.
Cowie, R., E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schroder, "Feeltrace: An instrument for recording perceived emotion in real time," Proceedings of ISCA Workshop on Speech and Emotion, (2000), 19-24.
Ekman, P., and W. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, 1978.
Hussain, A., E. Cambria, T. Mazzocco, M. Grassi, Q.-F. Wang, and T. Durrani, "Towards IMACA: Intelligent Multimodal Affective Conversational Agent," Lecture Notes in Computer Science, Vol. 7663(2012), 656-663.
Jung, M.-K., and J.-K. Kim, "The Intelligent Determination Model of Audience Emotion for Implementing Personalized Exhibition,"Journal of Intelligence and Information Systems, Vol. 18, No. 1(2012), 39-57.
Jung, M. K., J. K. Kim, Y. Ryu, and H. Ahn, "An intelligent determination model of user's emotion using sequential neural network," Proceedings of International Conference on Marketing Studies (ICMS 2012), (2012).
Kim, C. K., Studies of Exhibit's Theory and Method, National Science Museum, Academic Series 12, 1996.
Kim, S., E. Ryoo, M. K. Jung, J. K. . Kim, and H. Ahn, "Application of Support Vector Regression for Improving the Performance of the Emotion Prediction Model," Journal of Intelligence and Information Systems, Vol. 18, No. 3(2012), 185-202.
Ko, K.-E., D.-J. Shin, and K.-B. Sim, "Development of context awareness and service reasoning technique for handicapped people," Journal of Korean Institute of Intelligent Systems, Vol. 18, No. 4(2008), 512-517.

원문보기 상세보기
Lange, K., Numerical Analysis for Statisticians, Springer, 2010.
Nicolaou, M., H. Gunes, and M. Pantic, "Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space," IEEE Transactions on Affective Computing, Vol. 2, No. 2(2011), 92-105.

상세보기
Nicolle, J., V. Rapp, K. Bailly. L. Prevost, and M. Chetouani, "Robust Continuous Prediction of Human Emotions using Multiscale Dynamic Cues," Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI '12), (2012), 501-508.
Quesenberry, C. P. and N. P. Jewell, "Regression analysis based on stratified samples," Biometrika, Vol. 73, No. 3(1986), 605-614.

상세보기
Russell, J. A. "A Circumplex Model of Affect," Journal of Personality and Social Psychology, Vol. 39, No. 6(1980), 1161-1178.

상세보기
Ryoo, E. C., H. Ahn, and J. K. Kim, "The Audience Behavior-based Emotion Prediction Model for Personalized Service," Journal of Intelligence and Information Systems, Vol. 19, No. 2(2013), 73-85.
Sayre, S., "Multimedia that matters: Gallery-based technology and the museum visitor," First Monday, Vol. 10, No. 5(2005), doi:10. 5210/fm.v10i5.1244.
Wikipedia, Kinect, 2014. Available at http://en.wikipedia.org/wiki/Kinect (Accessed on January 15, 2014).
Wollmer, M., A. Metallinou, F. Eyben, B. Schuller, and S. Narayanan, "Context-Sensitive Multimodal Emotion Recognition from Speech and Facial Expression using Bidirectional LSTM Modeling," Proceedings of INTERSPEECH 2010, (2010), 2362-2365.
Wollmer, M., M. Kaiser, F. Eyben, F. Weninger, B. Schuller, and G. Rigoll, "Fully Automatic Audiovisual Emotion Recognition: Voice, Words, and the Face," Proceedings of 10. ITG Symposium, Speech Communication, (2012), 1-4.

저자의 다른 논문 :

LOADING...

표제어: PCR

동의어: Packet Collision Rate

용어 설명 출처 목록 (6)

용어 설명: PCR은 세균 특이성이 있는 primer를 이용하여 적은 수의 세균이 있을지라도 쉽게 검출할 수 있는 유용한 방법이며, 이를 이용하여 구강 내 치면세균막이나 타액에서 직접 세균을 검출할 수 있게 되었다[8].

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증