Leaked emails show the college board was given plenty of warning, which it could have used to avoid the recent MCQ exam failure.
ACRRM’s board knew about the potential of software failure during the computer-based fellowship exams, with leaked emails showing the college was warned almost a year prior to the recent failure of the Multiple-Choice Question exam.
In September, almost 60 registrars were asked by ACRRM to re-sit the MCQ exam following a glitch which prevented their test answers from being recorded correctly.
The computer-based exam was attempted on the day by almost 90 candidates in a supervised setting in their local communities.
And while ACRRM is conducting a review into the exam failure, it’s been suspected that the examination software, provided by testing company Questionmark, was faulty.
ACRRM was quick to apologise for the malfunction and promised to cover any costs associated with candidates sitting a second supervised MCQ. But the refunds issued to the ACRRM candidates (exam date 4 September) were only announced on 13 October, one day after refunds were announced for the RACGP exam candidates (scheduled exam dates 9 and 10 October).
TMR now understands that the recent exam complications may not have been entirely unexpected.
A recently leaked email chain between ACRRM staff and various members of the college executive reveals the concerns some examiners had about the effectiveness of delivering fellowship exams using the Questionmark platform.
Last month, ACRRM staff told college president Dr Ewen McPhee and president-elect Dr Sarah Chalmers, that they had expressed concerns about the delivery of the computer-based exams for consecutive years between 2016-2020.
The staff said they were not surprised when they heard of the glitch during the MCQ exam on 4 September, citing a series of technical issues in the five years prior with the same software.
The emails also linked to an independent review that ACRRM commissioned in October 2019 which reported on the performance of the Questionmark software.
In the recommendations, ACRRM was told that administrative staff and candidates had reported experiences of various technical glitches during the fellowship exams.
“In this regard, the software is unreliable for the delivery of consistent and fair assessment to all candidates,” the report said.
TMR put a series of questions to ACRRM about its knowledge of the potential for an exam malfunction, and whether it had taken appropriate steps to have a contingency plan in place on the day of assessments.
Dr McPhee told TMR that the college had always made considerable efforts to scope the nature of the exam format and product and consider the risks and mitigation strategies.
Questionmark was considered a known and reliable system, having been used twice a year by ACRRM, every year since 2007.
“Although there have been various minor technical glitches for some candidates over the years, the failure of the assessment on 4 September 2020 is the first time the college has experienced a system malfunction,” Dr McPhee said.
“None of the pre-tests or trials for the exam indicated any issues whatsoever.”
Instead, Dr McPhee said that ACRRM was aware that some of its examiners had been seeking to promote a different system, and had a view on exams which is contrary to the approach taken by the college.
“That was their personal view and preference. A number of assertions were made, but no evidence was provided to suggest that the Questionmark product was not reliable,” he said.
ACRRM said it had still heeded the examiners’ feedback, which is what led to the college’s independent review of the Questionmark product.
“The review findings also reveal that some ACRRM users (examinees, examiners, and administrative staff) of Questionmark perceived that it was not user-friendly because the interface was complex and not always easy or intuitive to use,” Dr McPhee said.
But ultimately, the platform was determined to be highly secure for the delivery of examinations to candidates at multiple sites, scheduling of assessments, and monitoring exams in real time.
“Obviously, in this year affected by COVID-19, the delivery of online assessments had added significance. The college increased its training and support for staff charged with implementing assessments using the Questionmark product,” Dr McPhee said.
“The college considered and planned for a possible system failure. It considered, given the nature of the assessment, that any significant extension of time or transfer to a paper-based back-up could place undue pressure on candidates, create unreliable results, and would not be a fair and reliable assessment of the candidates’ ability.
“The college determined that, in the event of a widespread malfunction, it would abandon that assessment, and re-schedule to another appropriate time. This is what occurred in September.”
Last week, ACRRM hosted the first of two re-sits scheduled for the MCQ assessment.
The second re-sit will take place on 21 November.
This story has been updated to include the date that refunds were announced for the candidates of the ACRRM MCQ exam.
