Share this article:

Hide

Recognising the need to future-proof and enhance their examination procedures the assessment team at the University College Cork’s School of Medicine identified that transitioning online was the way forward. As an existing Speedwell customer, they turned to us to help.

Established in 1849, University College Cork (UCC) has an impressive history spanning over 170 years, proudly ranking in the top 1.1% of universities worldwide[1]. UCC’s School of Medicine delivers world-class medical education providing future medical professionals with the knowledge and expertise to excel in their careers. The administration of robust and secure assessments is central to ensuring they’re well-prepared for the challenges of the field.

Dr John MacSharry, Deputy Director of the Graduate Entry to Medicine (GEM) programme explains that their examinations primarily take the form of multiple choice (MCQ) assessments in the first two years of the programme before they move onto practical assessments in the form of objective structured clinical exams (OSCE’s), in the latter years.

The School of Medicine historically delivered MCQ exams using Speedwell OMR sheets which would be scanned for marking and analysis. Item creation was completed by individual departments on Word documents, each team following its own processes.

Dr MacSharry explains, that after graduating students continuously train for another 5-10 years which often involves taking online exams. To best prepare students for this experience, he and Professor Mary Horgan, the Director of the GEM programme, felt that moving to online exams was important. This would mean that from the student’s perspective, their experience more closely simulates real-world practice.

It was clear that moving to an online format promised to save time and streamline processes in addition to enhancing the student experience. However, they were also mindful that such a significant change could lead to anxiety among students and faculty. They decided that a gradual approach would be best to bring everyone on board and ensure they were comfortable with the change.

Moving online – a gradual transition

It made sense to initially start the online transition by introducing a question banking system, before moving their MCQ exams online at a later date, with a further plan to extend to OSCEs.

It was crucial to find a suitable partner to meet their needs and facilitate the stepped move online. The team reviewed several platforms, however, only Speedwell could provide all of the comprehensive features they were looking for. Ultimately, they decided to move forward with the eSystem due to its robust question banking system, intuitive interface, additional question styles (for future proofing their exams) and compatibility with paper exams.

This enabled the School of Medicine to reap the benefits of question banking software while still conducting traditional paper exams. Furthermore, given their long-standing partnership with Speedwell, it was a logical decision.

Harnessing question banking benefits

Dr MacSharry explains that their process (before moving online) primarily consisted of each department having questions on Word documents. It was fraught with issues such as inconsistent formatting and password problems, which wasted a lot of time. Moreover, the external review of the questions was hampered by these issues, frustrating the external moderators because they were feeding back on formatting and were not able to just focus on question performance. It was embarrassing for the team. Since the adoption of the eSystem question bank, these issues have been completely removed. Now, each department simply adds its questions directly to the bank. Once happy they can set it to review and the moderators can simply log on and view the questions. The workflow feature assists the process by giving visibility and notifications to team members at each stage.

Initially, there was some internal resistance with concerns that the questions would be lost or it would take an enormous amount of time to get them into the bank. However, these fears were quickly allayed according to Dr MacSharry; “I gave a demo to my colleagues and initially there was some panic, but I was able to show them that you can just put the questions into a .csv and upload them. It takes less than 30 minutes. That stopped them in their tracks”

With that issue resolved it was hard to argue against the other benefits that the bank could bring, for example, greater transparency on question performance and version control. If an issue is discovered with an item, a new version can be created and edited. The complete performance history is available with each version being tracked with a full audit. This is incredibly useful and something that was not readily available before.

Implementing the eSystem question bank proved to be a great success. It’s improved processes and saved huge amounts of time and administration, whilst giving greater insight into question performance.

After using the eSystem question bank for several years alongside paper exams, the team felt confident to take the next step and look at moving their MCQ exams online.

Transforming assessments – the transition to online

In March 2020 the School of Medicine were preparing to deliver their first online MCQs. The intention was for students to take the exams in person on their own devices under invigilated conditions. However, the Covid-19 pandemic arrived with its subsequent lockdowns and their plans needed to adapt quickly. Dr MacSharry explains;

 “The exams were set up and ready in the eSystem when the Pandemic struck. This really helped, it enabled us to continue with the exams with minimal disruption”

Since they were no longer able to conduct in-person exams, it was decided that the exams would be run remotely, using secure exam browser. This ensured that students could not access anything else on their devices whilst taking the exam. This gave the team the confidence that the integrity of the exam could be protected and alleviated some internal fears over security and cheating.

Anticipating that there may be a degree of anxiety from students about a new exam format, the team took the time to create a few test exams. This allowed students to practice ahead of the actual exam. Once the students had trialled the system there was very little resistance; they found it easy to use and didn’t have any issues with the software.

Since a return to in-person exams has been allowed, the School of Medicine has continued to run exams online, they book a lecture theatre and have students come in and take the exam on their device using the lockdown browser, under invigilated conditions. They have a computer lab nearby so that should a student have a problem with their device they can move and still take the exam. Students only get the access password once they are in the exam hall. The exam invigilators have been pleased with the benefits online exams offer, Dr MacSharry says; “The invigilators are delighted. All they need to do now is make sure nobody is cheating. There are no carrying loads of papers, writing incident reports or preventing people from stealing exam papers – it’s made their job much easier”

He continues that the biggest benefits of using the eSystem are the speed and efficiency it offers the whole team:

“It makes our processes much more efficient, you can quickly create an exam from questions in the bank. Publish it, run it and have the results as soon as the exam is over. It’s very fast”

The School of Medicine’s exam process has undergone a complete transformation. Previously, exams were manually built, collecting questions from a series of Word documents and copying and pasting them into one paper, often encountering formatting issues. Once approved it needed to be sent off for print, along with answer sheets. After the exams were completed, an entire day would be dedicated to scanning answer sheets and addressing issues such as missed or faint marks. The scanned data would then go onto a USB stick and be organised before it could be analysed and statistics run. It was not unusual for this process to take several weeks.

However, using the eSystem things have changed dramatically. Now they can seamlessly organise, create, deliver and analyse exams in one central platform in a matter of days. A dramatic improvement.

Students are reaping the benefits too. Before it could take up to a month for results to be released back to students – much to their dissatisfaction. Now, it’s possible to get the results back the next day. Dr MacSharry explains:

 “We’re able to get results back the next day. The students are happier, they get their results quickly and they immediately know where they stand”   

Embracing the full potential

Looking ahead at what’s next for their assessment processes, Dr MacSharry explains that there is a desire to start using alternative question formats, specifically short answer questions and hotspot image questions.

This is partly due to the recognition that students can develop proficiency in MCQ exams, prompting the need for a more diversified approach, but also to help future-proof exams, especially in the context of emerging AI tools. The team feel that having more variable questions to test students’ depth of knowledge will be key in future examination practices.

For in-class continuous assessments, they are exploring using the feedback feature to increase and enhance feedback for students.

Longer term, they are eager to capitalise on the benefits that the eSystem can bring to their OSCE exams. The variety of question styles and the ability to cater for both written and practical assessments was a key factor behind the decision to choose Speedwell.

The School of Medicine have been pleased with what the eSystem offers so far, it’s revolutionised their exam processes and eliminated manual procedures and extensive time delays resulting in a significant improvement in efficiency and effectiveness.

[1] www.ucc.ie/en/discover/history/