[논문]반려묘의 상황인지형 행동 캡셔닝 시스템

채희찬; 최윤아; 이종욱; 박대희; 정용화

doi:10.9717/kmms.2020.24.1.021

반려묘의 상황인지형 행동 캡셔닝 시스템
Context-Awareness Cat Behavior Captioning System 원문보기

멀티미디어학회논문지 = Journal of Korea Multimedia Society, v.24 no.1, 2021년, pp.21 - 29

채희찬 (InfoValleyKorea) , 최윤아 (Dept of Computer and Information Science, Korea University) , 이종욱 (Dept of Computer and Convergence Software, Korea University) , 박대희 (Dept of Computer and Convergence Software, Korea University) , 정용화 (Dept of Computer and Convergence Software, Korea University)

Abstract ▼ AI-Helper

With the recent increase in the number of households raising pets, various engineering studies have been underway for pets. The final purpose of this study is to automatically generate situation-sensitive captions that can express implicit intentions based on the behavior and sound of cats by embedding the already mature behavioral detection technology of pets as basic element technology in the video capturing research. As a pilot project to this end, this paper proposes a high-level capturing system using optical-flow, RGB, and sound information of cat videos. That is, the proposed system uses video datasets collected in an actual breeding environment to extract feature vectors from the video and sound, then through hierarchical LSTM encoder and decoder, to identify the cat's behavior and its implicit intentions, and to perform learning to create context-sensitive captions. The performance of the proposed system was verified experimentally by utilizing video data collected in the environment where actual cats are raised.

주제어

표/그림 (12)

그림 Fig. 1. Overall structure of cat behavior captioning system.
그림 Fig. 2. Cat sound classification model.
그림 Fig. 3. Two-leveled hierarchical LSTM encoder.
그림 Fig. 4. Structure of the global/local decoder.
표 Table 1. Configuration of cat video clips.
그림 Fig. 5. Example of video clip annotations.
표 Table 2. Hyper-parameters of the proposed model.
표 Table 3. Quantitative results of HACA and proposed models.
그림 Fig. 6. Result of cat behavior captioning with and without using optical-flow, respectively.
그림 Fig. 7. Result of cat behavior captioning with and with-out using cat sounds, respectively.
표 Table 4. Two levels of cat behavior captioning with and without using cat sounds, respectively.
그림 Fig. 8. Captioning results of HACA and proposed models.

AI 본문요약
AI-Helper

제안 방법

본 연구의 실험에서는 영상의 시각 정보(RGB)와 객체의 움직임 정보(optical-flow), 그리고 객체의 소리 정보를 사용한 고수준의 상황인지형 캡셔닝 결과를 정성적, 정량적 성능 평가로 확인한다.

참고문헌 (26)

Pet Trend Report 2020. https://blog.opensurvey.co.kr/trendreport/companionanimal-2020 (accessed April 13, 2020).
P. Kumpulainen, A. Valldeoriola, S. Somppi, H. Tornqvist, H. vaataja, P. Majaranta, et al., "Dog Activity Classification with Movement Sensor Placed on the Collar," The Fifth International Conference on Animal-Computer Interaction, Vol. 4, pp. 1-6, 2018.
Y. Kim, J. Sa, Y, Chung, D. Park, and S. Lee, "Resource-Efficient Pet Dog Sound Events Classification Using LSTM-FCN Based on Time-Series Data," Sensors, Vol. 18, No. 11, 4019, 2018.

상세보기
Y. Pandeya, D. Kim, and J. Lee, "Domestic Cat Sound Classification Using Learned Features from Deep Neural Nets," Applied Sciences, Vol. 8, 1949, 2018.

상세보기
S. Lee and I. Kim, "Video Captioning with Visual and Semantic Features," Journal of Information Processing Systems, Vol. 14, No. 6, pp. 1318-1330, 2018.

원문보기 상세보기
P. Pan, Z. Xu, Y. Yang, F. Wu, and Y. Zhuang, "Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning," The IEEE Conference on Computer Vision and Pattern Recognition, pp. 1029-1038, 2016.
C. Ladha, N. Hammerla, E. Hughes, P. Olivier, and T. Ploetz, "Dog's Life: Wearable Activity Recognition for Dogs," Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 415-418, 2013.
L. Nanni, G. Maguolo, and M. Paci, "Data Augmentation Approaches for Improving Animal Audio Classification," Ecological Informatics, Vol. 57, 101084, 2020.

상세보기
M. Lee, J. Kang, and S. Lim, "Design of YOLO-Based Removable System for Pet Monitoring," Journal of the Korea Institute of Information and Communication Engineering, Vol. 24, No. 1, pp. 22-27, 2020.
X. Wang, Y. Wang, and W. Wang, "Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning," arXiv preprint arXiv:1804.05448, 2018.
J. Careira and Z. Andrew, "Quo Vadis, Action Recognition? a New Model and the Kinetics Dataset," The IEEE Conference on Computer Vision and Pattern Recognition, pp. 629-6308, 2017.
S. Hershey, S. Chaudhuri, D.P. Elis, J.F. Gemmeke, A. Jansen, R.C. More, M. Plakal, D. Plat, et al., "CNN Architectures for Large-Scale Audio Classification," 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 131-135, 2017.
D. Bahdanau, K. Cho, and Y. Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate," arXiv preprint arXiv: 1409.0473, 2014.
Q. Zhou and H. Wu, "NLPat IEST 2018: BiLSTM-Attention and LSTM-Attention via Soft Voting in Emotion Classification," Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 189-194, 2018.
Z. Huang, W. Yu, and K. Yu, "Bidirectional LSTM-CRF Models for Sequence Tagging," arXiv preprint arXiv:1508.01991, 2015.
M. Hong, H. Ahn, O. Atif, J. Lee, D. Park, and Y. Chung, "Field-Applicable Pig Anomaly Detection System Using Vocalization for Embedded Board Implementations," Applied Sciences, Vol. 10, No. 19, 6991, 2020.

상세보기
J. Seo, H. Ahn, D. Kim, S. Lee, Y. Chung, and D. Park, "EmbeddedPigDet: Fast and Accurate Pig Detection for Embedded Board Implementations," Applied Sciences, Vol. 10, No. 8, 2878, 2020.

상세보기
Y. Choi, O. Atif, J. Lee, D. Park, and Y. Chung, "Noise-Robust Sound-Event Classification System with Texture Analysis," Symmetry, Vol. 10, No. 9, 402, 2018.

상세보기
J. Lee, Y. Choi, Y. Chung, and D. Park, "Sound Noise-Robust Porcine Wasting Diseases Detection and Classification System Using Convolutional Neural Network," Journal of Korean Institute of Information Technology, Vol. 16, No. 5, pp. 1-13, 2018.

상세보기
S. Park, J. Kim, and D. Kim, "A Study on Classification Performance Analysis of Convolutional Neural Network Using Ensemble Learning Algorithm," Journal of Korea Multimedia Society, Vol. 22, No. 6, pp. 665-675, 2019.

원문보기 상세보기
H. Park, S. Bhattacharjee, P. Deekshitha, C. Kim, and H. Choi, "A Study on Deep Learning Binary Classification of Prostate Pathological Images Using Multiple Image Enhancement Technique," Journal of Korea Multimedia Society, Vol. 23, No. 4, pp. 539-548, 2020.
K. Jo, S. Jung, and C. Sim, "A Study of Shiitake Disease and Pest Image Analysis Based on Deep Learning," Journal of Korea Multimedia Society, Vol. 23, No. 1, pp. 50-57, 2020.
Raspberry Pi Official Website, https://projects.raspberrypi.org/en/projects/getting-started-with-picamera (accessed October 11, 2020).
X. Chen, H. Fang, T.Y. Lin, R. Vedantam, and S. Gupta, "Microsoft COCO Captions: Data Collection and Evaluation Server," arXiv preprint arXiv:1504.00325, 2015.
J.B Pam, "Think Like a Cat: How to Raise a Wel-Adjusted Cat-Not a Sour Pus," Pengui Books, 2017.
J.B Pam, "Cat Wise: America's Favorite Cat Expert Answers Your Cat Behavior Questions," Penguin Books, 2016

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증