[논문]생성형 인공지능의 수학 문제 풀이에 대한 성능 분석: ChatGPT 4, Claude 3 Opus, Gemini Advanced를 중심으로

오세준; 윤정은; 정유진; 조윤주; 심효섭; 권오남

doi:10.7468/mathedu.2024.63.3.549

[국내논문] 생성형 인공지능의 수학 문제 풀이에 대한 성능 분석: ChatGPT 4, Claude 3 Opus, Gemini Advanced를 중심으로
Analysis of generative AI's mathematical problem-solving performance: Focusing on ChatGPT 4, Claude 3 Opus, and Gemini Advanced 원문보기

Journal of the Korean Society of Mathematical Education. Series A. The Mathematical Education, v.63 no.3, 2024년, pp.549 - 571

오세준 (홍익대학교) , 윤정은 (인천 효성고등학교) , 정유진 (언주중학교) , 조윤주 (반송고등학교) , 심효섭 (구룡중학교) , 권오남 (서울대학교)

초록
AI-Helper

디지털·AI 기반 교수·학습이 강조됨에 따라 생성형 AI의 교육적 활용에 대한 논의가 활발해지고 있다. 본 연구는 고등학교 1학년 수학 교과서 5종의 예제와 문제 풀이에 대한 ChatGPT 4, Claude 3 Opus, Gemini Advanced의 수학적 성능을 분석하였다. 총 1,317개 문항에 대해 전체 정답률과 기능별 특징을 살펴본 결과, ChatGPT 4의 전체 정답률이 0.85로 가장 높았고, Claude 3 Opus가 0.67, Gemini Advanced가 0.42 순으로 나타났다. 기능별로는 함수 구하기와 증명하기에서 세 모델 모두 높은 정답률을 보였으나, 설명하기와 그래프 그리기에서는 상대적으로 낮은 정답률을 보였다. 특히 경우의 수 세기에서 ChatGPT 4와 Claude 3 Opus가 1.00의 정답률을 보인 반면, Gemini Advanced는 0.56으로 낮았다. 또한 모든 모델이 벤 다이어그램을 이용한 설명하기와 이미지 생성이 필요한 문제에서 어려움을 겪었다. 연구 결과를 바탕으로 교사들은 각 AI 모델의 강점과 한계를 파악하고 이를 수업에 적절히 활용할 수 있을 것이다. 본 연구는 생성형 AI의 수학적 성능을 분석함으로써, 실제 수학 수업에서의 생성형 AI의 활용 가능성을 제시했다는 점에서 의의가 있다. 또한 인공지능시대의 수학 교육에서 교사의 역할을 재정립하는 데 중요한 시사점을 제공하였다. 향후 생성형 AI와 교사의 협력적 교육 모델 개발, AI를 활용한 개별화 학습 방안 연구 등이 필요할 것이다.

Abstract ▼ AI-Helper

As digital·AI-based teaching and learning is emphasized, discussions on the educational use of generative AI are becoming more active. This study analyzed the mathematical performance of ChatGPT 4, Claude 3 Opus, and Gemini Advanced on solving examples and problems from five first-year high school math textbooks. As a result of examining the overall correct answer rate and characteristics of each skill for a total of 1,317 questions, ChatGPT 4 had the highest overall correct answer rate of 0.85, followed by Claude 3 Opus at 0.67, and Gemini Advanced at 0.42. By skills, all three models showed high correct answer rates in 'Find functions' and 'Prove', while relatively low correct answer rates in 'Explain' and 'Draw graphs'. In particular, in 'Count', ChatGPT 4 and Claude 3 Opus had a correct answer rate of 1.00, while Gemini Advanced was low at 0.56. Additionally, all models had difficulty in explaining using Venn diagrams and creating images. Based on the research results, teachers should identify the strengths and limitations of each AI model and use them appropriately in class. This study is significant in that it suggested the possibility of use in actual classes by analyzing the mathematical performance of generative AI. It also provided important implications for redefining the role of teachers in mathematics education in the era of artificial intelligence. Further research is needed to develop a cooperative educational model between generative AI and teachers and to study individualized learning plans using AI.

Keyword

표/그림 (21)

표 Table 1. Publishers, representative authors, and number of problems in five high school <Mathematics> textbooks
그림 Figure 1. Coding procedures by skills, problem type, accuracy rate.
표 Table 2. Contents system of the 2015 revised mathematics curriculum
그림 Figure 2. Procedure for deriving the accuracy rate using pivot table in Excel.
표 Table 3. Accuracy rates by content elements for answers of examples and problems in the high school <Mathematics> textbook by generative AI models
표 Table 4. Example of incorrect answer using ChatGPT 4 in factorization
표 Table 5. Accuracy rates by presence of image file for answers of examples and problems in the high school <Mathematics> textbook by generative AI models
표 Table 6. Example of problem including image with same context of text
표 Table 7. Example of problem including image only in question
표 Table 8. Example of problem where an image should be included in the solution
표 Table 9. Accuracy rates by skills for answers of examples and problems in the high school textbook by generative AI models
표 Table 10. Example of a problem that requires generating a Venn diagram image
표 Table 11. Example of a problem that requires finding functions
표 Table 12. Accuracy rates by problem types in proof problems
표 Table 13. Example of a problem requiring proof using the contrapositive statement
표 Table 14. Examples of correct and incorrect answers by Gemini Advanced for a counting problem using the rule of sum
표 Table 15. Example of a problem that requires drawing graphs
표 Table 16. Example of incorrect answer by Gemini Advanced in drawing graphs
표 Table 17. Accuracy rates by problem types in identifying problems
표 Table 18. Examples of incorrect and correct answers by differences in prompts in identifying problems
표 Table 19. Example of an incorrect answer to a problem identifying geometric figures

참고문헌 (39)

Ahn, D., Son, T., & Lee, K. (2023). ChatGPT as a scaffolding tool: Evaluating the impact on elementary students' mathematical？logic problem-solving skills. Brain, Digital, & Learning, 13(2), 183-196. https://doi.org/10.31216/BDL.20230011

상세보기
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P.,... & Amodei, D. (2020). Language models are？few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901. https://doi.org/10.48550/arXiv.2005.14165
Christudas, B. C. L., Kirubakaran, E., & Thangaiah, P. R. J. (2018). An evolutionary approach for personalization of content？delivery in e-learning systems based on learner behavior forcing compatibility of learning materials. Telematics and Informatics, 35 (3), 520-533. https://doi.org/10.1016/j.tele.2017.02.004

상세보기
CTOL Editors. (2024, April 16). Latest LLM market share Mar 2024: ChatGPT leads, Gemini surges, and Claude triples. CTOL Digital Solutions. https://www.ctol.digital/news/latest-llm-market-share-mar-2024-chatgpt-leads-geminisurges-and-claude-triples/
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E.,... & Strang, G. (2022). A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 119 (32), e2123433119. https://doi.org/10.48550/arXiv.2112.15594
Frieder, S., Pinchetti, L., Griffiths, R. R., Salvatori, T., Lukasiewicz, T., Petersen, P. C.,... & Berner, J. (2023). Mathematical？capabilities of ChatGPT. arXiv preprint arXiv:2301.13867. https://doi.org/10.48550/arXiv.2301.13867
Galatolo, F. A., Cimino, M. G., & Vaglini, G. (2022). Zero-shot mathematical problem solving via generative pre-trained？transformers. Proceedings of the 24th International Conference on Enterprise Information Systems (ICEIS 2022), 1, 479-483.
Go, B., Lim, C., & Shin, B. C. (2024). Development of a math-AI convergence instructional model using a generative AI？chatbot. Journal of Educational Technology, 40 (1), 1-40. http://dx.doi.org/10.17232/KSET.40.1.1

상세보기
Govender, R. (2023). The impact of artificial intelligence and the future of ChatGPT for mathematics teaching and learning？in schools and higher education. Pythagoras, 44 (1), 1-2. https://pythagoras.org.za/index.php/pythagoras/article/view/787
Hong, S., Lee, J., Shin, T., Lee, C., Lee, B., Shin, Y., ... & Kang, I. (2018). High school mathematics. Jihaksa.
Kang, Y. (2024). A study on the didactical application of ChatGPT for mathematical word problem solving. Communications？of Mathematical Education, 38 (1), 49-67. https://doi.org/10.7468/jksmee.2024.38.1.49

원문보기 상세보기
Kim, J. (2023). Leading teachers' perspective on teacher-AI collaboration in education. Education and Information Technologies, 1-32. https://doi.org/10.1007/s10639-023-12109-5

상세보기
Kim, W., Cho, M., Bang, G., Yoon, J., Shin, J., Im, S., ... & Jung, J. (2018). High school mathematics. Visang.
Ko, S., Lee, J., Lee, S., Cha, S., Kim, Y., Oh, T., & Cho, S. (2018). High school mathematics. Sinsago.
Kwon, O. N., Oh, S. J., Yoon, J., Lee, K., Shin, B. C., & Jung, W. (2023). Analyzing mathematical performances of ChatGPT:？Focusing on the solution of national assessment of educational achievement and the college scholastic ability test.？Communications of Mathematical Education, 37 (2), 233-256. http://doi.org/10.7468/jksmee.2023.37.2.233

원문보기 상세보기
Lee, J., Choi, B., Kim, D., Lee, J., Jeon, C., Chang, H., ... & Kim, M. (2018). High school mathematics. Chunjae.
Lee, S.G., Park, D., Lee, J.Y., Lim, D.S., Lee, J. H. (2024). Use of ChatGPT in college mathematics education. The Mathematical？Education, 63 (2), 123-138. https://doi.org/10.7468/MATHEDU.2024.63.2.123

원문보기 상세보기
Lee, Y. (2023). An analysis of pre-service teachers' mathematics lesson design using ChatGPT. Communications of Mathematical？Education, 37 (3), 497-516. https://doi.org/10.7468/jksmee.2023.37.3.497

원문보기 상세보기
Lv, H. Z. (2023). Innovative music education: Using an AI-based flipped classroom. Education and Information Technologies, 28 (11), 15301-15316.

상세보기
Mhlanga, D. (2023). Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. In FinTech？and artificial intelligence for sustainable development: The role of smart technologies in achieving development goals？(pp. 387-409). Springer. https://doi.org/10.1007/s10639-023-11835-0
Ministry of Education. (2015). Mathematics Curriculum. Bulletin of MOE No. 2015-74 [Separate Volume #8]. Ministry of Education.
Ministry of Education. (2022). Mathematics curriculum. Bulletin of MOE No.2022-33 [Separate Volume #8]. Ministry of Education.
Ministry of Education. (2024). Support plan for strengthening education innovation capacity based on digital technology.？Ministry of Education. https://www.moe.go.kr/boardCnts/viewRenew.do?boardID72769&boardSeq94551&lev0&searchTypenull&statusYNW&page1&smoe&m0315&opTypeN
Oh, S. J. (2023). Effective ChatGPT prompts in mathematical problem solving: Focusing on quadratic equations and？quadratic functions. Communications of Mathematical Education, 37 (3), 545-467. https://doi.org/10.7468/jksmee.2023.37.3.545

원문보기 상세보기
Oh, S. J. (2024). An analysis of the use of technology tools in high school mathematics textbooks based. Communications？of Mathematical Education, 38 (2), 263-286. https://doi.org/10.7468/JKSMEE.2024.38.2.263

원문보기 상세보기
Park, J., Song, M. Y., Nam, M., & Choi, K. (2021). Applying an integrated approach to analyse item features for curriculumbased？assessments. Journal of Curriculum Evaluation, 24 (1), 101-122. https://doi.org/10.29221/jce.2021.24.1.101

상세보기
Parra, V., Sureda, P., Corica, A., Schiaffino, S., & Godoy, D. (2024). Can generative AI solve geometry problems? Strengths？and weaknesses of LLMs for geometric reasoning in Spanish. International Journal of Interactive Multimedia and Artificial？Intelligence (IJIMAI), 8 (5), 65-74. https://dx.doi.org/10.9781/ijimai.2024.02.009

상세보기
Ryu, H., Sunwoo, H., Shin, B., Cho, J., Lee, B., Kim, Y., ... & Jung, S. (2018). High school mathematics. Chunjae.
Son, T. (2023). Exploring the possibility of using ChatGPT in mathematics education: Focusing on student product and？pre-service teachers' discourse related to fraction problems. Education of primary school mathematics, 26 (2), 99-113. https://doi.org/10.7468/jksmec.2023.26.2.99

원문보기 상세보기
Supriyadi, E., & Kuncoro, K. S. (2023). Exploring the future of mathematics teaching: Insight with ChatGPT. Union: Jurnal？Ilmiah Pendidikan Matematika, 11 (2), 305-316. http://dx.doi.org/10.30738/union.v11i2.14898
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,... & Polosukhin, I. (2017). Attention is all you？need. Advances in neural information processing systems, 30. https://doi.org/10.48550/arXiv.1706.03762
Vuorikari, R., Jerzak, N., Karpinski, Z., Pokropek, A., & Tudek, J. (2022). Measuring digital skills across the EU: Digital Skills？Indicator 2.0. Publication office of the European Union. https://data.europa.eu/doi/10.2760/897803
Wardat, Y., Tashtoush, M. A., AlAli, R., & Jarrah, A. M. (2023). ChatGPT: A revolutionary tool for teaching and learning？mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 19 (7), em2286. https://doi.org/10.29333/ejmste/13272

상세보기
Wong, T. K. Y., Tao, X., & Konishi, C. (2018). Teacher support in learning: Instrumental and appraisal support in relation to？math achievement. Issues in Educational Research, 28 (1), 202-219. http://www.iier.org.au/iier28/wong.pdf
Yoon, J., Park, S., & Kwon, O. N. (2023). ChatGPT-flipped mathematics class case study: Focused on learners' engagement.？Journal of Educational Technology, 39 (4), 1011-1047. http://dx.doi.org/10.17232/KSET.39.4.1011

상세보기
Zafrullah, Z., Hakim, M. L., & Angga, M. (2023). ChatGPT open AI: Analysis of mathematics education students learning？interest. Journal of Technology Global, 1 (01), 1-10.
Zhang, Z., Amiri, H., Liu, Z., Zufle, A., & Zhao, L. (2023). Large language models for spatial trajectory patterns mining. arXiv？preprint arXiv, 2310.04942. https://doi.org/10.48550/arXiv.2310.04942
Zhu, J. J., Jiang, J., Yang, M., & Ren, Z. J. (2023). ChatGPT and environmental research. Environmental Science & Technology, 57(46), 17667-17670. https://pubs.acs.org/doi/10.1021/acs.est.3c01818?gotosupporting-info

상세보기
Zhu, Y., & Yang, F. (2023). ChatGPT/aigc and educational innovation: Opportunities, challenges, and the future. Journal of？East China Normal University (Educational Sciences), 41 (7), 1. https://doi.org/10.16382/j.cnki.1000-5560.2023.07.001

저자의 다른 논문 :

활용도 분석정보

상세보기

다운로드

내보내기

활용도 Top5 논문

해당 논문의 주제분야에서 활용도가 높은 상위 5개 콘텐츠를 보여줍니다.
더보기 버튼을 클릭하시면 더 많은 관련자료를 살펴볼 수 있습니다.

표제어: PCR

동의어: Packet Collision Rate

용어 설명 출처 목록 (6)

용어 설명: PCR은 세균 특이성이 있는 primer를 이용하여 적은 수의 세균이 있을지라도 쉽게 검출할 수 있는 유용한 방법이며, 이를 이용하여 구강 내 치면세균막이나 타액에서 직접 세균을 검출할 수 있게 되었다[8].

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증