[논문]이미지 감성분류를 위한 CNN과 K-means RGB Cluster 이-단계 학습 방안

김정태; 박은비; 한기웅; 이정현; 이홍주

doi:10.13088/jiis.2021.27.3.139

이미지 감성분류를 위한 CNN과 K-means RGB Cluster 이-단계 학습 방안
A Two-Stage Learning Method of CNN and K-means RGB Cluster for Sentiment Classification of Images 원문보기

지능정보연구 = Journal of intelligence and information systems, v.27 no.3, 2021년, pp.139 - 156

김정태 (가톨릭대학교 수학과) , 박은비 (가톨릭대학교 수학과) , 한기웅 (고려대학교 산업경영공학과) , 이정현 (연세대학교 정보산업공학과) , 이홍주 (가톨릭대학교 경영학과)

초록
AI-Helper

이미지 분류에서 딥러닝 모형을 사용하는 가장 큰 이유는 이미지의 전체적인 정보에서 각 지역 특징을 추출하여 서로의 관계를 고려할 수 있기 때문이다. 하지만 이미지의 지역 특징이 없는 감정 이미지 데이터는 CNN 모델이 적합하지 않을 수 있다. 이러한 감정 이미지 분류의 어려움을 해결하기 위하여 매년 많은 연구자들이 감정 이미지에 적합한 CNN기반 아키텍처를 제시하고 있다. 색깔과 사람 감정간의 관계에 대한 연구들도 수행되었으며, 색깔에 따라 다른 감정이 유도된다는 결과들이 도출되었다. 딥러닝을 활용한 연구에서도 색깔정보를 활용하여 이미지 감성분류에 적용하는 연구들이 있어왔으며, 이미지만을 가지고 분류 모형을 학습한 경우보다 이미지의 색깔 정보를 추가로 활용한 경우가 이미지 감성 분류 정확도를 더 높일 수 있었다. 본 연구는 사람이 이미지의 감정을 분류하는 기준 중 많은 부분을 차지하는 색감을 이용하여 이미지 감성 분류 정확도를 향상시키는 방안을 제안한다. 이미지의 RGB 값에 K 평균 군집화 방안을 적용하여 이미지를 대표하는 색을 추출하여, 각 감성 클래스 별 해당 색깔이 나올 확률을 가중치 식으로 변형 후 CNN 모델의 최종 Layer에 적용하는 이-단계 학습방안을 구현하였다. 이미지 데이터는 6가지 감정으로 분류되는 Emotion6와 8가지 감정으로 분류되는 Artphoto를 사용하였다. 학습에 사용한 CNN 모델은 Densenet169, Mnasnet, Resnet101, Resnet152, Vgg19를 사용하였으며, 성능 평가는 5겹 교차검증으로 CNN 모델에 이-단계 학습 방안을 적용하여 전후 성과를 비교하였다. CNN 아키텍처만을 활용한 경우보다 색 속성에서 추출한 정보를 함께 사용하였을 때 더 좋은 분류 정확도를 보였다.

Abstract ▼ AI-Helper

The biggest reason for using a deep learning model in image classification is that it is possible to consider the relationship between each region by extracting each region's features from the overall information of the image. However, the CNN model may not be suitable for emotional image data without the image's regional features. To solve the difficulty of classifying emotion images, many researchers each year propose a CNN-based architecture suitable for emotion images. Studies on the relationship between color and human emotion were also conducted, and results were derived that different emotions are induced according to color. In studies using deep learning, there have been studies that apply color information to image subtraction classification. The case where the image's color information is additionally used than the case where the classification model is trained with only the image improves the accuracy of classifying image emotions. This study proposes two ways to increase the accuracy by incorporating the result value after the model classifies an image's emotion. Both methods improve accuracy by modifying the result value based on statistics using the color of the picture. When performing the test by finding the two-color combinations most distributed for all training data, the two-color combinations most distributed for each test data image were found. The result values were corrected according to the color combination distribution. This method weights the result value obtained after the model classifies an image's emotion by creating an expression based on the log function and the exponential function. Emotion6, classified into six emotions, and Artphoto classified into eight categories were used for the image data. Densenet169, Mnasnet, Resnet101, Resnet152, and Vgg19 architectures were used for the CNN model, and the performance evaluation was compared before and after applying the two-stage learning to the CNN model. Inspired by color psychology, which deals with the relationship between colors and emotions, when creating a model that classifies an image's sentiment, we studied how to improve accuracy by modifying the result values based on color. Sixteen colors were used: red, orange, yellow, green, blue, indigo, purple, turquoise, pink, magenta, brown, gray, silver, gold, white, and black. It has meaning. Using Scikit-learn's Clustering, the seven colors that are primarily distributed in the image are checked. Then, the RGB coordinate values of the colors from the image are compared with the RGB coordinate values of the 16 colors presented in the above data. That is, it was converted to the closest color. Suppose three or more color combinations are selected. In that case, too many color combinations occur, resulting in a problem in which the distribution is scattered, so a situation fewer influences the result value. Therefore, to solve this problem, two-color combinations were found and weighted to the model. Before training, the most distributed color combinations were found for all training data images. The distribution of color combinations for each class was stored in a Python dictionary format to be used during testing. During the test, the two-color combinations that are most distributed for each test data image are found. After that, we checked how the color combinations were distributed in the training data and corrected the result. We devised several equations to weight the result value from the model based on the extracted color as described above. The data set was randomly divided by 80:20, and the model was verified using 20% of the data as a test set. After splitting the remaining 80% of the data into five divisions to perform 5-fold cross-validation, the model was trained five times using different verification datasets. Finally, the performance was checked using the test dataset that was previously separated. Adam was used as the activation function, and the learning rate was set to 0.01. Th

주제어

참고문헌 (38)

Ali, A. R., U. Shahis, M. Ali, J. Ho, "High-Level Concepts for Affective Understanding of Images," arXiv:1705.02751v1, 2017, https://arxiv.org/abs/1705.02751
Argyris, Y. A., Z. Wang, Y. Kim, Z. Yin, "The effects of visual congruence on increasing consumers' brand engagement: An empirical investigation of influencer marketing on instagram using deep-learning algorithms for automatic image classification," Computers in Human Behavior, Vol. 112 (2020), 106443.

상세보기
Chen, M., L. Zhang, J. P. Allebach, "Learning deep features for image emotion classification," Proceedings of 2015 IEEE International Conference on Image Processing(ICIP), Quebec, Canada, 2015, 4491~4495.
Corchs, S., E. Fersini, F. Gasparini, "Ensemble learning on visual and textual data for social image emotion classification," International Journal of Machine Learning and Cybernetics, Vol. 10, No. 8 (2019), 2057~2070.

상세보기
Cruz, R. A., H. J. Lee, "The Brand Personality Effect: Communicating Brand Personality on Twitter and its Influence on Online Community Engagement," Journal of Intelligence and Information Systems, Vol. 20, No. 1 (2014), 67-101.

원문보기 상세보기
D'Andrade, R., M. Egan, "The colors of emotion," American Ethnologist, Vol. 1 (1974), 49-63.

상세보기
Ekman, P., "An argument for basic emotions," Cognition Emotion, Vol. 6 (1992), 169-200.

상세보기
Fei, Z., E. Yang, D. D. Li, S. Butler, W. Ijomah, X. Li, H. Zhou, "Deep convolution network based emotion analysis towards mental health care," Neurocomputing, Vol. 388 (2020), 212~227.

상세보기
Gajarla, V., A. Gupta, "Emotion detection and sentiment analysis of images," Georgia Institute of Technology, 2015.
Gilbert, A. N., A. J. Fridlund, L. A. Lucchina, "The color of emotion: A metric for implicit color associations," Food Quality and Preference, Vol. 52 (2016), 203~210.

상세보기
Gupta, S., S. K. Gupta, "Investigating Emotion-Color Association in Deep Neural Netwokrs," arXiv:2011.11058, 2020, https://arxiv.org/abs/2011.11058
Han, G.-W., J. H. Lee, H. J. Lee, "A CNN and K-means RGB Cluster Ensemble Method for Image Sentiment Classification," Proceedings of 2020 Spring Korea Intelligent Information Systems Society Conference, Seoul, South Korea, 2020, 26.
He, K., X. Zhang, S. Ren, J. Sun, "Deep Residual Learning for Image Recognition," Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, 770~778.
Kim, S. I., D. S. Kim, J. W. Kim, "Public Sentiment Analysis of Korean Top-10 Companies : Big Data Approach Using Multi-categorical Sentiment Lexicon," Journal of Intelligence and Information Systems, Vol. 22, No. 3 (2016), 45~69.

원문보기 상세보기
Lee, E., J. A. Lee, J. H. Moon, Y. Sung, "Pictures speak louder than words: Motivations for using Instagram," Cyberpsychology, behavior, and social networking, Vol. 18, No. 9 (2015), 552-556.

상세보기
Lee, J.-S., D. H. Park, "Development of Customer Sentiment Pattern Map for Webtoon Content Recommendation," Journal of Intelligence and Information Systems, Vol. 25, No. 4 (2019), 67-88.
Lee, J., Park, E., "Fuzzy Similarity-Based Emotional Classification of Color Images," IEEE Transactions on Multimedia, Vol. 13, No. 5 (2011), 1031-1039.

상세보기
Li, B., C. Guo, H. Ren, "Image Emotion Recognition Based on Deep Neural Network," Proceedings of 2018 IEEE International Conference of Safety Produce Informatization (IICSPI) (2018), 561~564.
Liao, S., J. Wang, R. Yu, K. Sato, Z. Cheng, "CNN for situations understanding based on sentiment analysis of twitter data," Procedia Computer Science, Vol. 111, 2017, 376-381.

상세보기
Liu, D., Y. Jiang, M. Pei, S. Liu, "Emotional image color transfer via deep learning," Pattern Recognition Letters, Vol. 110 (2018), 16~22.

상세보기
Nam, M., E. Lee, J. Shin, "A Method for User Sentiment Classification using Instagram Hashtags," Korea Multimedia Society, Vol. 18, No. 11 (2015), 391-399.
Netzer, O., R. Feldman, J. Goldenberg, M. Fresko, "Mine Your Own Business: Market-Structure Surveillance Through Text Mining," Marketing Science, Vol. 31, No. 3, (2012), 521-543.

상세보기
Machajdik, J., A. Hanbury, "Affective image classification using features inspired by psychology and art theory," Proceedings of the ACM Multimedia 2010 International Conference(MM' 10), Firenze, Italy, 2010, 83-92.
Mikels, J. A., B. L. Fredrickson, G. R. Larkin, C. M. Lindberg, S. J. Maglio, "Emotional category data on images from the international affective picture system," Behavior Research Methods, Vol. 37, No. 4 (2005), 626-630.

상세보기
Osgood, C. E., "The Cross-Cultural Generality of Visual-Verbal Synesthetic Tendencies," Behavioral Science, Vol. 5 (1960), 146-169.

상세보기
Panda, R. J. Zhang, H. Li, J.-Y. Lee, X. Lu, A. K. Roy-Chowdhury, "Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias," Proceedings of European Conference on Computer Vision (ECCV), Munich, Germany, 2018, 594~612.
Park, H. J., K. S. Shin, "Aspect-Based Sentiment Analysis Using BERT: Developing Aspect Category Sentiment Classification Models," Journal of Intelligence and Information Systems, Vol. 26, No. 4 (2020), 1-15.

원문보기 상세보기
Peng, K., T. Chen, A. Sadovnik and A. Gallagher, "A mixed bag of emotions: Model, predict, and transfer emotion distributions," Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, 860-868.
Priya, D. T., J. D. Udayan, "Affective emotion classification using feature vector of image based on visual concepts," The International Journal of Electrical Engineering & Education, (July 2020), 1~22.
Schlosberg, H., "Three dimensions of emotion," Psychological Review, Vol. 61, No. 2 (1954), 81-88.

상세보기
Seo, S.-H., J.-T. Kim, "Research trend of deep learning based sentiment analysis," Korea Multimedia Society, Vol. 20, No. 3 (2016), 8~22.
Song, K., T. Yao, Q. Ling, T. Mei, "Boosting image sentiment analysis with visual attention," Neurocomputing, Vol. 312 (2018), 218-228.

상세보기
Yang, Y., J. Jia, S. Zhang, B. Wu, Q. Chen, "How do your friends on social media disclose your emotions?" Proceedings of the National Conference on Artificial Intelligence, Quebec, Canada, 2014, 306-312.
Yang, J., M. Sun, X. Sun, "Learning visual sentiment distributions via augmented conditional probability neural network," Proceedings of AAAI Conference on Artificial Intelligence, San Francisco, California, USA, 2017, 224~230.
Zhang, W., X. He, W. Lu, "Exploring Discriminative Representations for Image Emotion Recognition With CNNs," IEEE Transactions on Multimedia, Vol. 22, No. 2 (2020), 515~523.

상세보기
Zhang, J., H. Sun, Z. Wang, T. Ruan, "Another Dimension: Towards Multi-subnet Neural Network for Image Sentiment Analysis," Proceedings of 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China, 2019, 1126-1131.
Zhao, S., G. Ding, Q. Huang, T.-S. Chua, B. W. Schuller, K. Keutzer, "Affective Image Content Analysis: A Comprehensive Survey," Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), Stockholm, Sweden, 2018, 5534-5541.
Understanding the Meaning of Colors in ColorPsychology, 2009. Available at http://www.empower-yourself-with-color-psychology.com/

저자의 다른 논문 :

표제어: PCR

동의어: Packet Collision Rate

용어 설명 출처 목록 (6)

용어 설명: PCR은 세균 특이성이 있는 primer를 이용하여 적은 수의 세균이 있을지라도 쉽게 검출할 수 있는 유용한 방법이며, 이를 이용하여 구강 내 치면세균막이나 타액에서 직접 세균을 검출할 수 있게 되었다[8].

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증