Share this article:

Hide

Responsible for overseeing the standard of Ophthalmology training for the European Union of Medical Specialists (UEMS) the European Board of Ophthalmology’s (EBO) diploma exam is highly regarded with applicants steadily increasing each year. Looking to modernise and future-proof this growing exam they turned to Speedwell to help.

The EBO Diploma is designed to assess the knowledge and clinical skills essential to the delivery of a high standard of ophthalmologic care both in hospitals and in independent clinical practices. It’s a two-part exam consisting of an MCQ paper and a Viva Voce interview, held in Paris each year.

The EBO has been a Speedwell customer for many years, and in 2015, they began looking to streamline their exam processes. At the time, each exam paper was marked by hand which was incredibly time-consuming and increasingly unfeasible with growing student numbers. With 550 candidates due to sit the exam, they needed a provider who could deliver a seamless solution that combined design, scanning and analysis of the exam and Speedwell was that partner. The level of service and support offered by Speedwell was a key deciding factor, having a technical consultant attend and assist during the exam was very important. The transition went well and the EBO was delighted; the move to Optical Mark Recognition (OMR) sheets and scanning allowed them to meet the increasing demand for the EBO exam and turn results around within hours.

Since then demand for the EBO exam has continued to grow each year with candidates from all over Europe taking the exam. This, coupled with the fact that the system used for the Viva Voce exams became outdated, prompted the Board to consider how to further streamline and future-proof the exam. They needed a solution that would allow the exam to quickly scale up, whilst mitigating any increased administration. It was becoming apparent that moving the exam online may be a better and more viable option.

A scalable, flexible system

Brendan Strong, a Project Manager from Agenda Communications and Conferencing, works closely with the EBO to help them run the exam each year, provides more context; “Presently, we regularly have 700 candidates apply for the exam and there’s a waiting list, not everyone gets in” He continues to explain that 2020 was the 25th anniversary of the EBO and to celebrate they planned to run two exams to accommodate more students. One in the spring in Paris and one in the autumn in Berlin. This would allow a significant increase in candidates taking the exam; “Running the two exams, we’d be going from 700 to 900 candidates and I think we could reasonably expect a 1000  candidates to be on the horizon in a few years.” He says. Having a system in place that would be scalable and could support this growing exam was key.

Furthermore, the solution needed to be flexible and that was a big consideration in the decision-making process. Any online system would need to be able to cope with the EBO’s unique MCQ marking system; for the MCQ exam, a negative mark (-0.5) is applied to each incorrect, multiple marked answer and those left blank. Choosing ‘don’t know’ results in a 0 score and one point is awarded for correct answers. The aim of the marking scheme to discourage ‘wild’ guessing.

Whilst this was not a problem for Speedwell’s eSystem, not all online systems could accommodate this. Although moving to the eSystem may seem to be the natural progression due to the existing relationship, Brendan wanted to highlight that when evaluating online solutions, it was this, coupled with the excellent customer support that set Speedwell apart;

“What made Speedwell stand out was the flexibility of the system, the confidence that it could be tailored to meet the needs of the EBO Exam and the great customer service and support offered.”

Unfortunately, the Covid-19 pandemic prevented the EBO from running the two exams as they had intended in 2020 and they were cancelled. However, all was not lost, they had been planning for tablets to be used in the exams. So, when Covid hit they felt confident that they could do something– although, like the rest of the world, exactly what was not immediately clear.

It transpired that it was possible to run some very small exams, whilst adhering to the Covid regulations in Switzerland. Laptops were rented, which could be used for both the MCQ and the Viva Voce exams. For the Viva Voce,  one examiner was present (socially distanced from the candidate they were interviewing). The laptops displayed presentations showing clinical data and images, but also connected a second examiner via video conferencing – so the interview process was similar to what would normally occur, albeit more remote. Moving the exams online enabled the EBO to deliver the exam in very challenging circumstances.

The online difference

Ahead of the first online exam, students and examiners were understandably nervous about how the online exam would look and feel. The EBO decided to create a dummy exam and give training so that students and examiners had the opportunity to familiarise themselves with the system before the actual exam.

Ultimately these reservations were unfounded, what they discovered was that the system was very easy to use for everyone and it was positively received. In particular, the EBO were happy that with the exam timing it’s easy to add extra time for candidates if needed. For example, in instances of connection problems, drop-outs or problems logging in, it meant that students were not penalised as extra time could easily be added to the exam so candidates still got their full 2 hours.

For the Viva Voce element of the exam, expectations were exceeded. Brendan describes the reaction of many of the  examiners;

“The examiners just felt it was so straightforward, everyone found it very easy to use, it was very well received.”

Another huge plus of the online system was the ability to monitor the exam in live; “The real-time monitoring of the exam was really helpful, we could clearly see if we had missing candidates or if there were any marking anomalies – these were picked up immediately and addressed at the time” A tangible improvement on the paper system where such situations wouldn’t become apparent until after the exam and then be time-consuming to follow-up and resolve.

Additionally, with an online system, you get the benefit of record-keeping. With eSystem, you have a secure, online database that is backed up in the cloud. With a paper exam there is always the risk that it could get lost or damaged, with online there’s much less risk, it negates the need for storage facilities and it can be accessed at any time quickly and easily by anyone with permission.

The EBO also found that gains were made in the marking process, with the paper exams, it could take up to 15–20  minutes to mark. All of the papers would have to be collected and scanned before you could get your results. That would be the best-case scenario without there being any problems, such as feint or missing marks – which could delay the process further. However, using eSystem the results are immediately available and you get certainty in  the answers, it’s absolutely clear what someone intends.

Brendan also thinks that candidates benefit as well;

“I think it’s easier for candidates – at the end of the test they  get a list of all of their answered questions and so they can see anything they have missed and can go back and answer”

He continues that it’s so easy to miss an answer with paper, but with eSystem, this problem is removed. Aside from the administrative efficiencies and streamlining of processes, the support offered by Speedwell is highlighted as an important factor in the relationship that has continued with the move to online exams.

Comprehensive, reliable support

Before they purchased the eSystem, Speedwell supported the paper EBO diploma exam by sending a technician to be on-site on the day of the exam. Their role to help scan papers, be on hand to quickly resolve any issues and ensure a quick turnaround of results. Brendan has high praise for the team;

“Customer support is very important to us and the support we have always received from Speedwell is excellent.”

He explains that with the move to online, Speedwell has helped at every step, from answering initial questions right through to being available remotely on the actual day of the exam; “I’d say a word of thanks goes to your support team” He continues “on the day of the exam they e-mailed to say they would be online and available to assist in the event of any problems. Then, at the end of the day, they emailed to say ‘we hope it went well and glad we didn’t hear from you!’ it was great to know that the support was there if we needed it.”

This support has been across the board, for the 2021 exam the EBO is considering changing the format of the Viva Voce exam. Keen to keep the exam online they wanted to know what options were available to them with the eSystem. They wanted to avoid the clinical cases being tested via MCQs because the consensus was that they should be more challenging. They also wanted to avoid paragraphs of text for the answers. With 700+ candidates that would just make the marking process too cumbersome.

So, the EBO Board met with the Speedwell development team who were able to discuss some options. It transpires that two different question styles would be appropriate. One is the Hotspot, which would allow a candidate to click on an image to identify a problem. The other is the Very Short Answer (VSA) question style. This allows candidates to freely type their answer, but that answer is limited to 2 or 3 words. Acceptable answers can be predefined in the question and so the exam can be automatically marked.

The team were relieved to learn the eSystem could accommodate the new format and allow the Viva Voce element of the exam to be modernised. “It’s great the eSystem is allowing us to make this change. That it has the flexibility and functionality within it” Brendan said. Having the meeting with Speedwell really made a difference. Afterwards, the exam board were much more confident with the fully online proposal. It was clear on how to move forward and design the exam so that it’s still in the same spirit and quality as the traditional exam.

The future

With 2020 acting as a small pilot for the online EBO exam, it’s given the board the confidence that it can work well. They are looking to continue with online exams in the future. For the 2021 exams, they are expecting around 1,000 candidates over two exams to clear the backlog from 2020 and assess new candidates applying in 2021 for the first time. Due to the ongoing uncertainty over Covid, the plan is to run these exams remotely and online. However, the availability of remote examiners is unsure. So, once again, the flexibility offered by Speedwell and the eSystem means that a solution could be found with the very short-answer question type.

Speedwell has also assisted the EBO in identifying and working with a remote-proctoring service, something they have not previously had to worry about.

As the Covid pandemic dissipates, the future is still uncertain. The EBO live exams were historically held with large ophthalmology meetings. Questions remain over whether meetings will be the same in the future, and if there will be space available to accommodate such a large exam like the EBO. However, the EBO have full confidence that whatever the future holds, Speedwell will be able to assist in finding a solution.

 

Case Study: European Board of Ophthalmology