[논문]BIM 운용 전문가 시험을 통한 ChatGPT의 BIM 분야 전문 지식 수준 평가

최지원; 구본상; 유영수; 정유정; 함남혁

doi:10.13161/kibim.2023.13.3.021

[국내논문] BIM 운용 전문가 시험을 통한 ChatGPT의 BIM 분야 전문 지식 수준 평가
Evaluating ChatGPT's Competency in BIM Related Knowledge via the Korean BIM Expertise Exam 원문보기

Journal of KIBIM = 한국BIM학회논문집, v.13 no.3, 2023년, pp.21 - 29

최지원 (서울과학기술대학교 건설시스템공학과) , 구본상 (서울과학기술대학교 건설시스템공학과) , 유영수 (서울과학기술대학교 건설시스템공학과) , 정유정 (서울과학기술대학교 건설시스템공학과) , 함남혁 (한양사이버대학교 디지털건축도시공학과)

Abstract ▼ AI-Helper

ChatGPT, a chatbot based on GPT large language models, has gained immense popularity among the general public as well as domain professionals. To assess its proficiency in specialized fields, ChatGPT was tested on mainstream exams like the bar exam and medical licensing tests. This study evaluated ChatGPT's ability to answer questions related to Building Information Modeling (BIM) by testing it on Korea's BIM expertise exam, focusing primarily on multiple-choice problems. Both GPT-3.5 and GPT-4 were tested by prompting them to provide the correct answers to three years' worth of exams, totaling 150 questions. The results showed that both versions passed the test with average scores of 68 and 85, respectively. GPT-4 performed particularly well in categories related to 'BIM software' and 'Smart Construction technology'. However, it did not fare well in 'BIM applications'. Both versions were more proficient with short-answer choices than with sentence-length answers. Additionally, GPT-4 struggled with questions related to BIM policies and regulations specific to the Korean industry. Such limitations might be addressed by using tools like LangChain, which allow for feeding domain-specific documents to customize ChatGPT's responses. These advancements are anticipated to enhance ChatGPT's utility as a virtual assistant for BIM education and modeling automation.

주제어

표/그림 (13)

그림 Figure 1. Three steps of method for reinforcement learning(Introducing ChatGPT (2022, November 30) OpenAI: https://openai.com/blog/chatgpt)
그림 Figure 2. GPT performance on academic and professional exams(OpenAI, 2023)
표 Table 1. Problem categories for the BIM expertise exam
그림 Figure 3. Distribution of questions by type (average)
그림 Figure 4. Example of short-answer/essay question
그림 Figure 5. Percentage of short-answer/essay question (average)
그림 Figure 6. ChatGPT results by year
표 Table 2. ChatGPT results by year
그림 Figure 7. Performance of ChatGPT by question type
그림 Figure 8. Example of ChatGPT correct answers
그림 Figure 9. Examples of ChatGPT incorrect answers
그림 Figure 10. ChatGPT performance with respect to short-answer vs. essay questions
그림 Figure 11. Example of ChatGPT correct/incorrect answers

참고문헌 (12)

Beetz, J., Van Leeuwen, J., De Vries, B. (2009). IfcOWL :？A Case of Transforming EXPRESS Schemas into Ontologies. Ai Edam, 23(1), pp. 89-101.？

상세보기
Bommarito II, M., Katz, D. M. (2022). GPT Takes the Bar Exam.？arXivpreprint arXiv:2212.14402.？
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.,？Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G. Askell,？A., Agarwal, S., Herbert-Voss, A. Krueger, G, Henighan,？T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C.,？Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess,？B., Clark, J., Berner, C., McCandlish, S., Radford, A.,？Sutskever, I., Amodei, D. (2020).Language Models are？Few-Shot Learners. Advances in Neural Information？Processing Systems, 33, pp. 1877-1901.？
Choi, J. H., Hickman, K. E., Monahan, A., Schwarcz, D. (2023).？ChatGPT Goes to Law School. Available at SSRN.？
Kasai, J., Kasai, Y., Sakaguchi, K., Yamada, Y., Radev, D.？(2023). Evaluating GPT-4 and ChatGPT on Japanese-Medical Licensing Examinations. JMIR Medical Education,？9.？
KIBIM (2023). A study on countermeasures for construction？engineering in accordance with the digitalization of the？construction industry, Final Report.？
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepano, C., Madriaga, M., Aggabao, R., DiazCandido, G., Maningo, J., Tseng, V. (2023). Performance？of ChatGPT on USMLE: Potential for AI-assisted Medical？Education Using Large Language Models. PLOS Digital？Health, 2(2).？
Mazairac, W., Beetz, J. (2013). BIMQL-An Open Query language？for Building Information Models. Advanced Engineering？Informatics, 27(4), pp. 444-456.？

상세보기
OpenAI (2023). GPT-4 Technical Report. ArXiv, abs/2303.08774.？
Seo, M. B., Ju, K. B. (2012). Strategies to Revitalize BIM？(Building Information Modeling) by the Survey Question-Naires from Design Experts in Field of Civil Engineering.？Journal of the Korea Contents Association, 12(11), pp.？446-457.？
Singh, A. K., Pal, A., Kumar, P., Lin, J. J., Hsieh, S. H. (2023).？Prospects of Integrating BIM and NLP for Automatic？Construction Schedule Management. ISARC. Proceedings？of the International Symposium on Automation and？Robotics in Construction. IAARC Publications, 40, pp.？238-245.？
Zheng, J., Fischer, M. (2023). BIM-GPT: A Prompt-based？Virtual Assistant Framework for BIM Information Retrieval. arXiv preprint, DOI: 10.48550/arXiv.2304.09333

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증