Adapting transformers to wording-based item difficulty prediction

Abstract

Accurate item difficulty estimation is a fundamental challenge in high-stakes educational test construction. Traditionally, difficulty is estimated through small-scale pretesting or expert judgments. However, pilot samples may not accurately represent the target population, estimates may exhibit large errors, and expert ratings are not sufficiently reliable. Alternatively, difficulty can be derived from item wording, but most approaches use various text features extracted from extensively preprocessed wording, leading to a loss of crucial morphosyntactic information. To address this, recent research (Netík et al., 2024) has explored fine-tuning transformer models that enable circumventing the preprocessing step, yet improvements over feature-based methods have been only marginal, presumably due to insufficient item wording representation. This paper introduces architectural modifications to improve model performance by ensuring a more comprehensive representation of the item. We propose three key changes: (1) a multitask learning framework that jointly predicts item difficulty and classifies wording components to encourage balanced attention distribution, (2) an encoding strategy that processes wording parts separately before merging them into a unified representation, and (3) leveraging deeper layers of the encoder stack instead of relying solely on the last hidden states of the first token. Additionally, we compare these odifications with the few-shot learning capabilities of state-of-the-art large language models and explore the potential of using their predictions to generate training data for fine-tuning smaller models. We aim to contribute to a more robust transformer-driven solution for item difficulty prediction, potentially reducing reliance on established yet limited approaches while improving the precision of difficulty estimates.