The use of multiple-choice tests (multiple choice questions (MCQ) and computer-based tests (CBT)) in internal and EPSO competitions has come under fire [1] due to concerns over their effectiveness and fairness. These tests, which are widely used to assess candidates’ knowledge and skills, have been found to have several pitfalls, including technical issues, translation problems, and, above all, poorly designed questions. Not to mention the apparent lack of any quality control before and after. For our own in-depth analysis of these matters see here. [2]
Wins and losses
One of the main advantages of multiple-choice tests is that they are easy to scale up and can be administered to many candidates at a relatively low cost. Additionally, the results are objective, as they are generated automatically, eliminating the need for human markers. This makes them an attractive option for organisations looking to streamline their hiring processes.
Alas, the possible advantages of MCQs have been more than overshadowed by the experience with MCQs in recent competitions, which has not necessarily been positive – to put it mildly. And here we are not only talking about technical issues such as frozen screens or faulty connections. Such pitfalls can occur regardless of the type of online test, while others are more inherent to the nature of the test.
Issues which could have been anticipated?
For instance, the fact that questions must be translated into multiple languages invariably leads to issues with nuances in translation, particularly in verbal reasoning questions: this is nothing new in the world of MCQ and multinational companies. Translations can and do result in questions being ambiguous or confusing, which can unfairly disadvantage candidates when such issues are not spotted early enough or addressed appropriately when they are eventually raised.
Furthermore, there have been competitions where a disproportionate number of questions have had to be cancelled, most likely due to errors or ambiguities. This not only undermines the credibility of the test but also raises questions about the quality of the questions and the testing process.
Fit for purpose?
In addition to these issues, there are concerns about the reliability of multiple-choice tests, particularly in specialist competitions. The tests are designed to measure a candidate’s knowledge and skills in a specific area, but the number of questions is way too low to cover the field. This can lead to candidates being selected based on chance rather than their actual abilities. Generation 2004 is concerned this can have serious consequences, not only for the candidates but also for the Commission using these tests to select future staff.
The lack of any quality control in the testing process is also a salient issue. Questions are not thoroughly tested before being administered to candidates, and there is no ex-post review of the results to determine whether the test was well-designed and effective in identifying the best candidates. As our analysis highlights, this lack of quality control can lead to a range of problems, from biased questions to, ultimately, unfair outcomes.
Are any lessons being learned?
In this context, our analysis also emphasises the importance of statistical analysis in evaluating the effectiveness of multiple-choice tests. By quantitatively analysing the results of the test, it is possible to identify areas where the test may be biased or unfair, and to make adjustments to improve the test’s validity and reliability. However, there is no evidence that such type of analysis is conducted, even where tests are administered repeatedly.
In conclusion, while multiple choice tests may have some advantages, their use in competitions is fraught with problems. The issues with translation, question design, and reliability, combined with the lack of quality control, raise serious concerns about the fairness and effectiveness of these tests. It is therefore essential that these concerns are addressed thoroughly and effectively. By doing so, the Commission can ensure that they are selecting the most qualified candidates for the job, and that the testing process is transparent and accountable.
If you appreciate our work, please consider becoming a member of Generation 2004 [3].