INNOVATING OSCEs: How the Arab Board Set New Standards in Medical Certification

CASE STUDY: ARAB BOARD OF HEALTH SPECIALIZATIONS – DIGITAL OSCE

Recognising the need to modernise and standardise medical assessments across the Arab world, the Arab Board of Health Specializations set out to transition their clinical exams to a digital format. With a vision for a seamless, transparent and efficient process, they turned to Speedwell to help.

The Arab Board of Health Specializations is a well-respected institution dedicated to enhancing medical education across the Arab world. Founded in 1978, its mission is to ensure consistent, high standards for medical professionals in the region.

Core to this mission is ensuring that all assessments are conducted in a standard, fair, equal, and transparent manner, a responsibility overseen by Professor Sabeeh Mashhadani in his role as Assessment Director.

Assuming the role following a distinguished career as a consultant dermatologist, Professor Mashhadani also brought extensive experience in curriculum design and assessment from his tenure at Baghdad College of Medicine. This experience, coupled with his passion for technology meant he was perfectly placed to innovate and modernise the exams.

Upon joining the Arab Board, one of the first things he noted was that the assessments were conducted using the traditional pen-and-paper method which had its limitations. Typically, there is one exam for all Arab Board candidates in a particular speciality, sometimes with the exam being delivered in different locations concurrently. He quickly identified that it was a challenge to administer standardised exams across multiple countries with this method.

Additionally, there were issues such as a lack of centrally stored questions, no historical data for improving future exams and inconsistent testing environments. He further noted that providing candidate feedback was cumbersome and time-consuming, highlighting further inefficiencies with this traditional approach.

Professor Mashhadani knew that change was necessary. He recognised the benefits of online exams and has been instrumental in championing the transition to digital assessments. The goal being to streamline operations and enhance the overall efficiency and quality of the examination processes of the Arab Board.

Professor Sabeeh Mashhadani

SELECTING THE RIGHT PARTNER FOR THE ARAB BOARD’S DIGITAL OSCEs

The digital transformation of exams started with the implementation of online written knowledge exams. Keen to build on this, attention soon shifted to how to utilise the latest examination technologies to enhance OSCE assessments.

However, identifying the right partner for this task proved challenging. Many online examination platforms lacked the capability to manage the intricate requirements of OSCE exams.

After evaluating options, Professor Mashhadani knew Speedwell would be the right choice to help him deliver his vision:

“Speedwell’s eSystem offers a complete, easy-to-implement solution that’s intuitive and user-friendly. It provides exactly what we need to transform our OSCE assessments.”

Not only is the eSystem a web-based platform that requires no local installation, but Speedwell also brought extensive experience in the medical education sector. With a proven track record of helping institutions successfully run OSCEs in various formats, on paper, online and offline over many years. Speedwell demonstrated a deep understanding of the complexities associated with practical clinical exams. This proven expertise and experience provided Professor Masshadani with the confidence that the eSystem was the right choice to modernise their assessments.

Furthermore, the software offers a robust solution that functions reliably with or without Internet connectivity. The latter was particularly important to Professor Mashhadani and his team as connections can be unreliable in parts of the Arab region at times.

NAVIGATING THE TRANSITION – THE PATH TO DIGITAL OSCEs

Despite having confidence in their chosen partner, the path to delivering online OSCEs was not without challenges.

Introducing a new assessment format typically generates considerable concern among stakeholders, with resistance to change being a primary obstacle, and the Arab Board was no exception. Professor Mashhadani believes that a fear of technology also played a significant role in his experience.

He explains that many of those involved belonged to a generation that did not grow up with technology. Understandably, there were reservations about learning to use new systems, coupled with a general lack of awareness about the benefits of transitioning to digital OSCEs. Additionally, there were concerns about the reliability of digital systems and their ability to maintain the integrity and rigour of the examinations.

This generation is accustomed to pen-and-paper exams and has vast experience running OSCEs in this way. Convincing them of the benefits of changing their tried and tested practices was no easy feat.

Addressing these concerns required a strategic approach involving comprehensive training, a robust support system and clear communication of the advantages to build confidence in the new format.

TRAINING FOR CHANGE – EMBRACING DIGITAL OSCEs

The Arab Board comprises more than 40 scientific councils, covering fields such as dermatology, general medicine, and surgery. Training such a large number of professionals initially seemed daunting. During early discussions, there were understandable concerns from stakeholders about the perceived complexity of the new system and the potential learning curve.

However, Professor Mashhadani was determined and with support and comprehensive training from Speedwell, he became an expert in the software. He personally undertook the training of all his colleagues, demonstrating the eSystem’s ease of use at each stage.

Key to his success was understanding his audience and teaching them in a manner that made them feel comfortable. He explains he focused on simplifying the training, concentrating on core areas relevant to the individual’s role to avoid overwhelming them with the system’s full capabilities.

Without exception, once people started using the system, their initial concerns disappeared – they quickly grasped how it works and were impressed by all of the benefits it offers. Professor Mashhadani explains; “Even the examiners who did not think they could learn the software, like using the eSystem once they use it”.

With training complete, it was time to put all of their hard work into practice.

RUNNING THE FIRST DIGITAL OSCE

The General Surgery Council led the way with the transition to digital OSCEs. Beginning with a successful pilot that provided confidence to proceed to a full-scale exam. It was decided that the exam would be delivered offline, with Professor Mashhadani assisting with the station set-up as well as monitoring the exam as it was in progress. It was delivered successfully and everyone was delighted with how well the exam went:

“We’ve achieved remarkable results with our first digital OSCE – quick, instant results and amazing statistical insights. It’s created a new standard for our exams.”

Professor Mashhadani continues, that before using the eSystem, the council would have to manually do a lot of work post-exam. They would use Excel to calculate the difficulty, discrimination and borderline regression. It could take the team several days to complete.

Now, using the eSystem, they’re amazed that they can instantly complete these tasks with the click of a button. Furthermore, decisions can be made about the test immediately after the exam which is a huge tangible difference. They now have the insight to improve stations and deliver better exams in the future.

When it comes to candidate feedback, the eSystem is delivering substantial benefits too. Previously, they could only provide feedback for each station; it wasn’t feasible to offer detailed insights. Now, they can offer two types of feedback: station-specific and competency-based.

For instance, a competency like differential diagnosis, which frequently recurs across stations, can now be specifically addressed. Candidates can be informed whether they excel in this competency or need improvement, in addition to receiving feedback for each station.

Professor Mashhadani emphasises the importance of this approach:

“Sometimes a candidate might pass a station but consistently underperform in a particular area, such as differential diagnosis. The eSystem enables us to identify and remedy any common areas of weakness”

He underscores that this is a significant advancement for them, as it allows for more nuanced and constructive feedback than before.

The General Surgery Council were so impressed by the benefits that the eSystem brought to their OSCEs, that they advocated its use to other councils. This endorsement has led to seven additional councils adopting Speedwell for their OSCEs to date with more likely in the near future.

Students waiting to start their OSCE

IMPLEMENTATION OF THE ARAB BOARD DIGITAL OSCE MANAGEMENT SYSTEM

The challenge didn’t end there. The clinical exams are held in exam centres accredited by the Arab Board. This is very important to ensure that all exams are conducted in a standard, fair and transparent manner.

One Council, the Family Medicine Council, deliver the clinical exam simultaneously in test centres in different countries. However historically there was no connection between the centres, they would all conduct the exams independently with no monitoring between them.

Professor Mashhadani started considering how he could use technology and the Speedwell eSystem to conduct a monitored exam in multiple countries simultaneously. The same exam, with the same stations, delivered at the same time.

In collaboration with the Dubai Health Authority and the General Secretariat of the Arab Board, a clinical examination was conducted using Speedwell’s eSystem for 126 doctors across five accredited test centres in various Arab countries. This exam, monitored through a centralised control centre within the health authority, was seamlessly administered with a high level of transparency and was deemed a huge success.

This event marked a historic first for the Arab Board, as it was the first instance of exams being conducted in this manner. Not only did Speedwell’s eSystem bring all of the benefits of streamlined exam management, but it was also central in the facilitation of the exam process, whilst ensuring the integrity of the assessment was upheld.

This accomplishment represents a significant milestone for the Arab Board, setting a new benchmark for future clinical assessments in the region.

SETTING A NEW STANDARD: LEADING THE FUTURE OF MEDICAL EDUCATION

Professor Mashhadani’s vision extends beyond the current achievements. He aims to expand the Digital OSCE Management System to more councils within the Arab Board, hoping to have 12 scientific councils using it by the end of 2024 and the remaining councils the following year. He believes that it’s a necessity to prepare for the digital era and explains;

“It’s not a choice if we do not go through the digital transformation, we will not achieve our goals”

His advocacy is not limited to the Arab Board. He hopes to integrate similar practices into pharmacy schools and undergraduate medical education as well as influence global practices outside of the Arab region. Ultimately, promoting a shift towards more efficient, transparent and standardised assessment methods.

For Professor Mashhadani and the Arab Board, the experience of transitioning to digital OSCEs with Speedwell has highlighted the importance of embracing technology to meet the evolving needs of medical education. While simultaneously illustrating how its transformative impact can significantly enhance the quality and efficiency of medical training.

Together, we have set a new benchmark for medical certification in the Arab region, ensuring that the next generation of healthcare professionals are evaluated with precision, fairness and efficiency.