Share this article:

Hide

Image courtesy of UCL

Committed to changing the world for the better, University College London (UCL) has long been at the forefront of innovative thinking. Established in 1826, it immediately provided an open education in England to students of any race, class or religion. And in 1878 this was extended to welcoming female students on equal terms with men.

It is this culture of pushing forward for the long term benefit of humanity that has seen the UCL Medical School advance the discovery of adrenaline and understanding of the immune system and autoimmune disease. No less than six former students and staff have been awarded Nobel Prizes.

With this outstanding reputation to protect, it was never going to be an easy decision about when to invest in a new question bank and exam software for their Medical School. But the imperative to gain operational efficiencies was becoming too hard to ignore.

Strategic considerations

From the outset of the project to find a replacement for their aging question bank software, the Medical School was quite clear on their priorities and constraints.  They wanted to ensure that the cohort starting at the school in Autumn 2017 would have the benefits of student centred feedback on exam performance as well as the assessment team being able to achieve operational gains.

The premium associated with lab space in the city, meant that UCL Medical School were realistic about the physical infrastructure required to host exams online for 400 students concurrently. It is much easier to hire the space required and continue to run paper exams – but everything else associated with running exams needed to be online.

Exam administration bottlenecks

Chloe Marshall leads the team of administrators responsible for delivering the written assessments for each cohort of 400 students across the 6 year MBBS programme at UCL.  Knowing that the School had built up a bank of over 5000 questions and experiencing first hand where the bottlenecks in the exam management processes for the team, meant that Chloe had a clear vision of what any new exam management software would need to achieve.

The layout of the existing question bank was causing headaches for Chloe’s team. When it was originally formatted some years ago, it made sense to take a modular approach to its design and to lay it out by academic year. But as the curriculum has developed, the School needed a more integrated approach to their bank structure that would allow them to easily manage questions and blueprint their exams across year groups.

From a practical perspective, Chloe was also focused on reducing errors and rework in the question bank and exam management process. Their existing software was installed on one or two machines in the team – and this meant that academics and question writers sent in their changes to questions and exams via email or as word documents.  This process was open to greater error and rework than Chloe was happy to tolerate.

Investing time to get the basics right

The decision to invest in eSystem software and use it alongside their existing MultiQuest software meant that UCL could get all the advanced banking benefits they were after as well as being able to continue to deliver paper exams.  Chloe’s extensive time investment in the project focused largely on setting up the new bank exactly as they needed – she lead the development of the bank structure using groups and subgroups as well as rolling over the valuable questions and performance history from their existing bank.

As Chloe said “The investment in the new question bank software had to represent a stepped change for us in terms of operational efficiencies. But we had to balance that with retaining enough of the historical data and statistics on each question and exam that would make the new bank immediately useful. We didn’t have the luxury of time building up our question statistics over time.”

Chloe took a pragmatic approach to rolling in the questions and their outting statistics – recreating all the exams over the past three years and importing the candidate results via MultiQuest into the eSystem. Chloe comments: “Importing the questions and their statistical history was a relatively easy to do – the investment came in terms of my time. But this was totally worth it. And, it was actually quite easy in the end.”

A future proofed investment

Even though Chloe doesn’t believe that UCL will be running online MCQ exams in the very near future, she is quite happy with the operational gains they have achieved using eSystem software and having the option to easily extend to online exam delivery in the future. For now, Chloe’s team scans the completed paper exam sheets into MultiQuest and then imports all the data into the eSystem. It is the eSystem that generates all the vital question statistics and prepares the student centred reports that her students keenly await after each exam. Chloe comments “Students want more and more feedback. And whilst we can’t give them the questions and answers, we can give them feedback reports – which they seem to really value. They can see at a glance how they perform by curriculum area. ”

Gaynor Jones, Head of MBBS Management at UCL is ultimately responsible for delivery of all assessments at the Medical School and is keen to ensure future needs of the School’s assessments are met.

“We have been a Speedwell customer for many years. And what I appreciate about working with Speedwell, is that they invest in their software to ensure it continues to meet the needs of their customers. With an open approach to getting feedback on what we needed, they have developed software that really gives us the best of both worlds – operational efficiencies, whilst continuing to deliver paper exams. And we can easily extend our license to online MCQ and OSCE exam delivery too. This flexible approach to their software design really sets Speedwell apart from other exam software suppliers.”

In 2018 UCL extended their eSystem license so that they could make the move from paper to digital OSCEs. Read the case study here.