[논문]종이에서 컴퓨터로의 매체 전환에 따른 평가 결과의 변화 탐색: 고등학교 수학 국가수준 학업성취도 평가를 중심으로

정혜윤; 송창근; 김영준; 이경화

doi:10.7468/mathedu.2022.61.4.597

종이에서 컴퓨터로의 매체 전환에 따른 평가 결과의 변화 탐색: 고등학교 수학 국가수준 학업성취도 평가를 중심으로
Exploring the change in achievement by the transition of the test mode from paper to computer: Focusing on the national assessment of educational achievement of high school mathematics 원문보기

Journal of the Korean Society of Mathematical Education. Series A. The Mathematical Education, v.61 no.4, 2022년, pp.597 - 612

정혜윤 (한국교육과정평가원) , 송창근 (서울대학교 대학원) , 김영준 (서울대학교 대학원) , 이경화 (서울대학교)

초록
AI-Helper

최근 국내외 대규모 평가를 중심으로, 수학 평가가 전통적인 지필 평가에서 컴퓨터 기반 평가로 전환되고 있다. 본 연구는 평가 문항의 유형, 문항에 반영된 기술공학적 기능, 학생의 컴퓨터 사용 특성 및 컴퓨터 평가 환경에 따라 평가 매체 전환에 따른 학생의 성취도 차이가 어떻게 변화하는지 탐색하였다. 이를 위해 지필 환경과 컴퓨터 환경에서 실행된 2020 고등학교 수학의 국가수준 학업성취도 평가 결과를 분석하였다. 연구결과, 첫째, 단순 매체만 전환된 경우 모드 효과는 대체로 미미하였지만, 서술형 문항의 경우 단답형이나 선택형 문항에 비해 모드 효과가 더 크게 나타났다. 둘째, 매체 전환과 기술공학적 기능의 반영이 함께 이루어진 경우, 문항에 반영된 기술공학적 기능에 따라 모드 효과에 차이가 있었다. 셋째, 컴퓨터 기반 평가 시행환경, 학생의 컴퓨터 활용 빈도와는 달리, 학생의 컴퓨터 활용 효능감에 따라 컴퓨터 기반 평가에서 통계적으로 유의한 점수 차이가 나타났다. 이와 같은 연구 결과는 향후 평가 내용과 역량에 맞춘 적절한 기술공학적 기능의 도입이 필요하며, 학생의 수학 학습 능력 외의 부수적인 능력이나 상황이 평가 결과에 영향을 미치지 않도록 세밀한 평가 설계와 환경 구비가 필요함을 시사한다.

Abstract ▼ AI-Helper

Recently, large-scale mathematics assessments are shifting from traditional paper-based tests to computer-based tests, nationally and internationally. This study explored the mode effect (the difference in student achievement by the change of test mode) according to the types of test items, the technological function reflected in the items, the characteristics of students' computer use, and the computer-based test environment. To this end, we analyzed the results of the 2020 national assessment of educational achievement of high school mathematics conducted on a paper and computer basis. As a result, firstly, the mode effect induced by the mode transition was generally insignificant, but the mode effect was larger in the extended response type than other types. Secondly, there were differences in the mode effect according to the transition to test with computer mode where innovative items were added. Thirdly, the difference between mode effects was statistically significant according to the student's sense of efficacy in computer use. The results of this study suggest that innovative items should be introduced deliberately according to the targeted content and competency in evaluation, and that assessment design and environment preparation need to be carefully developed so that nonessential abilities other than students' mathematical ability or incidental situation do not distort the assessment results.

주제어

표/그림 (12)

그림 Figure 1. An example of drag and drop item (Arslan et al., 2020)
그림 Figure 2. An example of pull-down item (Scalise & Gifford, 2006)
표 Table 1. The frequency in the computer use, the sense of efficacy in computer use, and the environment for CBT
표 Table 2. The item types presented in PBT, IBT, and CBT
그림 Figure 3. The correct answer rate of each item presented in PBT and IBT (unit: %)
그림 Figure 4. The average of the correct answer rate for PBT and IBT (unit: %) and the difference in the average of the correct answer rate for PBT and IBT (unit: %p) by the item type
표 Table 3. Item type and the correct answer rate for PBT and IBT of each item
그림 Figure 5. The correct answer rate of each item presented in PBT and CBT (unit: %)
표 Table 4. The correct answer rate for PBT and CBT of each item and technological function reflected in CBT
표 Table 5. The difference in the achievement according to the frequency of the computer use
표 Table 6. The difference in the achievement according to the environment for CBT
표 Table 7. The difference in the achievement according to the sense of efficacy in computer use

참고문헌 (48)

Arslan, B., Jiang, Y., Keehner, M., Gong, T., Katz, I. R., & Yan, F. (2020). The Effect of Drag-and-Drop Item Features on Test-Taker Performance and Response Strategies. Educational Measurement: Issues and Practice, 39(2), 96-106. https://doi.org/10.1111/emip.12326

상세보기
Backes, B. & Cowan, J. (2019). Is the pen mightier than the keyboard? The effect of online testing on measured student achievement. Economics of Education Review, 68(1), 89-103. https://doi.org/10.1016/j.econedurev.2018.12.007

상세보기
Bennett, R. E., Braswell, J., Oranje, A., Sandene, B., Kaplan, B., & Yan, F. (2008). Does it Matter if I Take My Mathematics Test on Computer? A Second Empirical Study of Mode Effects in NAEP. Journal of Technology, Learning, and Assessment, 6(9), 1-38.
Bridgeman, B. (1992). A Comparison of quantitative questions in open-ended and multiple-choice formats. Journal of Educational Measurement, 29(3), 253-271.

상세보기
Buerger, S., Kroehne, U., Koehler, C., & Goldhammer, F. (2019). What makes the difference? The impact of item properties on mode effects in reading assessments. Studies in Educational Evaluation, 62(3), 1-9. http://dx.doi.org/10.1016/j.stueduc.2019.04.005

상세보기
Clements, D. H. (2020). From exercises and tasks to problems and projects: Unique contributions of computers to innovative mathematics education. The Journal of Mathematical Behavior, 19(1), 9-47. http://dx.doi.org/10.1016/S0732-3123(00)00036-5

상세보기
Csapo, B., Ainley, J., Bennett, R. E., Latour, T., & Law, N. (2012). Technological issues for computer-based assessment. In P. Griffin, B. McGaw, & E. Care (Eds.). Assessment and teaching of 21st century skills. Springer.
Drasgow, F., & Mattern, K. (2006). New tests and new Items: Opportunities and issues. In D. Bartram, & R. K. Hambleton (Eds.), Computer-based Testing and the Internet: Issues and Advances (pp. 59-76). John Wiley and Sons Ltd.
Ebrahimi, M.R., Toroujeni, S.M., & Shahbazi, V. (2019). Score equivalence, gender difference, and testing mode preference in a comparative study between computer-based testing and paper-based testing. International Journal of Emerging Technologies in Learning, 14(7), 128-143. https://doi.org/10.3991/ijet.v14i07.10175

상세보기
Eid, G. K. (2005). An investigation into the effects and factors influencing computer-based online math problem-solving in primary schools. Journal of Educational Technology Systems, 33(3), 223-240. https://doi.org/10.2190/J3Q5-BAA5-2L62-AEY3

상세보기
Fishbein, B., Martin, M. O., Mullis, I. V., & Foy, P. (2018). The TIMSS 2019 item equivalence study: Examining mode effects for computer-based assessment and implications for measuring trends. Large-scale Assessments in Education, 6(1), 1-23. https://doi.org/10.1186/s40536-018-0064-z

상세보기
Geraniou, E., & Jankvist, U. T. (2019). Towards a definition of "mathematical digital competency". Educational Studies in Mathematics, 102(1), 29-45. https://doi.org/10.1007/s10649-019-09893-8

상세보기
Haladyna, T., Downing, S., & Rodriguez, M. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-333. https://doi.org/10.1207/S15324818AME1503_5

상세보기
Heyd-Metzuyanim, E., Sharon, A. J., & Baram-Tsabari, A. (2021). Mathematical media literacy in the COVID-19 pandemic and its relation to school mathematics education. Educational Studies in Mathematics, 108(1), 201-225. https://doi.org/10.1007/s10649-021-10075-8

상세보기
Hoogland, K., & Tout, D. (2018). Computer-based assessment of mathematics into the twenty-first century: pressures and tensions. ZDM, 50(4), 675-686. https://doi.org/10.1007/S11858-018-0944-2

상세보기
Horkay, N., Bennett, R., Allen, N., Kaplan, B., &Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. The Journal of Technology, Learning, and Assessment, 5(2). Retrieved from https://ejournals.bc.edu/index.php/jtla/article/view/1641
Hu, L., Chen, G., Li, P., & Huang, J. (2021). Multimedia effect in problem solving: A meta-analysis. Educational Psychology Review, 33(4), 1717-1747. https://psycnet.apa.org/doi/10.1007/s10648-021-09610-z

상세보기
Jeong, H. (2014). A comparative study of scores on computer-based tests and paper-based tests. Behaviour & Information Technology, 33(4), 410-422. https://doi.org/10.1080/0144929X.2012.710647

상세보기
Jewsbury, P., Finnegan, R., Xi, N., Jia, Y., Rust, K., Burg, S., Donahue, P., Mazzeo, J., Cramer, E. B., & Lin, A. (2017). NAEP transition to digitally based assessments in mathematics and reading at grades 4 and 8: Mode evaluation study. National Center for Education Statistics.
Jung, H. Y. (2022). Perceptions of students and teachers towards computer based test in National Assessment of Educational Achievement : Focused on high school mathematics test. School Mathematics, 24(1), 119-145. http://doi.org/10.29275/sm.2022.3.24.1.119
Karay, Y., Schauber, S., Stosch, C., & Schuttpelz-Brauns, K. (2015). Computer versus paper: Does it make any difference in test performance? Teaching and Learning in Medicine, 27(1), 57-62. https://doi.org/10.1080/10401334.2014.979175

상세보기
Keng, L., McClarty, K., & Davis, L. (2008). Item-level comparative analysis of online and paper administrations of the texas assessment of knowledge and skills. Applied Measurement in Education, 21(3), 207-226. https://doi.org/10.1080/08957340802161774

상세보기
Kingston, N. (2008). Comparability of computer- and paper-administered multiple-choice tests for K-12 populations: A synthesis. Applied Measurement in Education, 22(1), 22-37. https://doi.org/10.1080/08957340802558326

상세보기
Lee, J. B., Kim, Y. H., Kim, J. S., Nam, M. W., Park, J. S., Park, J. H., Baek, J. H., Sung, K. H., Lee, S. R., Jang, E. S., & Jung, H. Y. (2021). Advancing the test tools for Electronic National Assessment of Educational Achievement (eNAEA). KICE.
Lee, S.-G., Ham, Y., Lee, J. H., & Park, K.-E. (2020). A case study on college mathematics education in untact DT era. Communications of Mathematical Education, 34(3), 201-214. https://doi.org/10.7468/JKSMEE.2020.34.3.201

원문보기 상세보기
Martin, M. O., von Davier, M., & Mullis, I. V. S. (2020) Methods and Procedures: TIMSS 2019 Technical Report. TIMSS & PIRLS International Study Center.
Martin, R. (2008). New possibilities and challenges for assessment through theuse of technology. In F. Scheuermann, & G. Pereira (Eds.). Towards a research agenda on computer-based assessment (pp. 6-9). Office for Official Publications of the European Communities.
Mcclelland, T., & Cuevas, J. (2020). A comparison of computer based testing and paper and pencil testing in mathematics assessment. The Online Journal of New Horizons in Education, 10(2), 78-89.
McDonald, A. (2002). The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments. Computers and Education, 39(3), 299-312. https://doi.org/10.1016/S0360-1315(02)00032-5

상세보기
Noyes, J., & Garland, K. (2008). Computer- vs. paper-based tasks: Are they equivalent? Ergonomics, 51(9), 1352-1375. https://doi.org/10.1080/00140130802170387

상세보기
Park, M. (2020). Applications and possibilities of artificial intelligence in mathematics education. Communications of Mathematical Education, 34(4), 545-561. https://doi.org/10.7468/JKSMEE.2020.34.4.545

원문보기 상세보기
Parshall, C. G., Harmes, J. C., Davey, T., & Pashley, P. J. (2009). Innovative items for computerized testing. In W. J. van der Linden (Ed.). Elements of adaptive testing (pp. 215-230). Springer. https://doi.org/10.1007/978-0-387-85461-8_11
Poggio, J., Glasnapp, D., Yang, X., & Poggio, A. (2005). A comparative evaluation of score results from computerized and paper & pencil mathematics testing in a large scale state assessment program. The Journal of Technology, Learning, and Assessment, 3(6).
Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: mode effects for passage-based tests. The Journal of Technology, Learning, and Assessment, 2(6), 1-44.
Rose, J., Low-Choy, S., Singh, P., & Vasco, D. (2020). NAPLAN discourses: A systematic review after the first decade. Discourse: Studies in the Cultural Politics of Education, 41(6), 871-886. https://doi.org/10.1080/01596306.2018.1557111

상세보기
Russell, M. (1999). Testing on computers: A follow-up study comparing performance on computer and on paper. Boston College.
Russell, M., & Tao, W. (2004). The influence of computer-print on rater scores. Practical Assessment, Research, and Evaluation, 9(10), 1-13.
Sandene, B., Horkay, N., Bennett, R., Allen, N., Braswell, J., Kaplan, B., & Oranje, A. (2005). Online assessment in mathematics and writing: Reports from the NAEP technology-based assessment project, research and development series. National Center for Education Statistics.
Scalise, K., & Gifford, B. (2006). Computer-based assessment in E-Learning: A framework for constructing "Internet Constraint" questions and tasks for technology platforms. The Journal of Technology, Learning, and Assessment, 4(6). Retrieved from https://www.learntechlib.org/p/103254/.
Schleppegrell, M. J. (2007). The linguistic challenges of mathematics teaching and learning: A research review. Reading & Writing Quarterly, 23(2), 139-159. https://doi.org/10.1080/10573560601158461

상세보기
Schoenfeld, A. H. (2017). On learning and assessment. Assessment in Education: Principles, Policy & Practice, 24(3), 369-378. https://doi.org/10.1080/0969594X.2017.1336986

상세보기
Shacham, M. (1998). Computer-based exams in undergraduate engineering courses. Computer Applications in Engineering Education, 6(3), 201-209.

상세보기
Sireci, S. G., & Zenisky, A. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representation. In S. M. Downing, &T. M. Haladyna (Eds.), Handbook of test development (pp. 329-347). Lawrence Erlbaum Associates Publishers.
Spires, H. A., Paul, C. M., & Kerkhoff, S. N. (2019). Digital literacy for the 21st century. In D. B. A. Khosrow-Pou (Ed.). Advanced methodologies and technologies in library science, information management, and scholarly inquiry (pp. 12-21). IGI Global.
Taylor, C., Jamieson, J., Eignor, D. & Kirsch, I. (1998). The Relationship between computer familiarity and performance on computer-based TOEFL test tasks. Educational Testing Service.
Yoo, S. M. (2013). SPSS statistical analysis for writing a thesis. Slow& Steady.
Wang, S., Jiao, H., Young, M. J., Brooks, T., & Olson, J. (2007). A meta-analysis of testing mode effects in grade K-12 mathematics tests. Educational and Psychological Measurement, 67(2), 219-238. https://doi.org/10.1177/0013164406288166

상세보기
Zilles, C., West, M., Mussulman, D., & Bretl, T. (2018). Making testing less trying: Lessons learned from operating a Computer-Based Testing Facility. In 2018 IEEE Frontiers in Education (FIE) Conference, IEEE.

표제어: PCR

동의어: Packet Collision Rate

용어 설명 출처 목록 (6)

용어 설명: PCR은 세균 특이성이 있는 primer를 이용하여 적은 수의 세균이 있을지라도 쉽게 검출할 수 있는 유용한 방법이며, 이를 이용하여 구강 내 치면세균막이나 타액에서 직접 세균을 검출할 수 있게 되었다[8].

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format