Predicting item difficulty with text analysis and machine learning in different languages and item types

Abstract

ZÁKLADNÍ ÚDAJE: IMPS 2024 Abstracts. Prague: IMPS, 2024. s. 309-309. [IMPS 2024: Annual Meeting of the Psychometric Society. 16.07.2024-19.07.2024, Prague]. ABSTRAKT: In standardized testing, predicting item difficulty from item wording is useful both for test development as well as for deeper understanding of what makes an item a difficult one. Many features may influence item difficulty, such as the length of answer choices, their similarity with the item question, difficulty of the words used, etc., and different machine learning models may be used to predict item difficulty from item features (Štěpánek et al., 2023). However, differences and challenges may arise when building models for different item types (including those involving audio, or visual components), and for different languages. In this work, we extract item features from various types of test items from the English, German, and French as foreign languages Czech matura exams into various item features, and train numerous different machine learning models to predict their difficulty. We compare and analyze the models and features in order to create a tool that can analyze and suggest changes during test development to help achieve an optimal item difficulty.