Give educators the skills to bring assessment into the future

In higher education, the demand for robust assessment systems has never been higher. We recently conducted a one-week graduate-level course on classic test theory (CTT) and item response theory (IRT) for STEM experts. CTT is a conventional statistical framework. It underpins conventional student testing that assists test developers to judge the general quality and their tests and questions. IRT is a more modern statistical framework, which enables test developers to implement cutting-edge and highly reliable assessment systems to monitor student progress. 

Here’s how we designed and delivered the course using an innovative approach, how we used practical examples, group work and visualisations and how we addressed the risk of committing ecological and atomistic fallacies when examining patterns in data, using a multi-level modelling component.

Think about your course structure

The course began with an exploration of classical test theory, providing a solid foundation for understanding the basics of test construction, reliability and validity, allowing participants to grasp fundamental concepts before delving into the more advanced aspects of assessment theory.

• Questions to test students’ understanding of research methods
• Four steps for integrating generative AI in learning and teaching
• ChatGPT and generative AI: 25 applications in teaching and assessment

We then transitioned to item response theory, introducing a modern approach to examining and scoring test results. Using practical examples involving real-world data, we made the theoretical concepts more tangible and equipped participants with practical skills in applying IRT to diverse assessment scenarios. 

Offer innovative pedagogical techniques

To enhance engagement and facilitate deeper learning, the course relied on innovative pedagogical techniques. Group work was a central component, providing participants with opportunities to collaboratively analyse and solve assessment challenges. This fostered teamwork and encouraged healthy debate between those with different perspectives and experiences. 

Use visualisations to teach complex concepts

Graphical representations of test characteristics, such as item characteristic curves and test information functions, enabled participants to visualise the underlying mathematics of the paradigm. This approach, combining theory with practical application, resonated well with the STEM experts, who often find visualisations to be powerful aids in understanding abstract concepts.

Extend the learning horizon

By introducing Rasch modelling, a popular approach for IRT, we gave participants the ability to identify students’ zones of proximal development, providing a nuanced understanding of individual learning trajectories. This approach transcends traditional assessments by offering insights into students’ potential growth areas, enabling educators to tailor interventions and support more effectively.

Addressing ecological and atomistic fallacies

In the final multi-level modelling component of the course, the focus shifted towards addressing ecological and atomistic fallacies. The tendency for educators to make ecological fallacies, which involves incorrectly making inferences about individuals based on group-level data, was discerned and countered through the exploration of multi-level modelling techniques. 

Similarly, the atomistic fallacy, which involves incorrectly making inferences about groups based on individual-level data, was challenged using the same modelling approach. We guided participants through the intricacies of multi-level models by combining formula with data frames and spreadsheets, allowing them to comprehend the complexities of aggregating and disaggregating data at different levels of analysis.

What real-world applications can your technology offer?

Throughout the course, we chose to integrate real-world examples and case studies to bridge the gap between theory and practice. Participants were encouraged to apply CTT and IRT principles to their own assessment scenarios and own real-world student selection exam data, giving them a sense of ownership and relevance. This enhanced their understanding and equipped them with the tools to implement changes in educational settings.

Consider the societal implications

The course delved into the broader societal implications of advanced standardised assessments. Recognising the need for trust between citizens and governments, particularly in the context of education, participants explored the role of sophisticated assessment systems in building transparency and accountability. 

Assessments not only ensure a fair and accurate representation of students’ capabilities but also contribute to fostering public trust in the education system, aligning educational outcomes with societal expectations.

This course on classical and modern test theory transcended traditional teaching methods to offer an engaging learning experience. The pedagogical techniques we incorporated provided a comprehensive foundation of skills and knowledge for developing and implementing assessment systems.

As the demand for state-of-the-art assessment systems continues to grow in schools and institutes of higher education, initiatives like these serve as beacons of innovation, guiding educators towards a future where assessments are not just tests but powerful tools for enhancing learning outcomes.

Matthew Courtney is the head of institutional research & analytics at the Office of the Provost at Nazarbayev University.

If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.