Share this article:

Hide

How can you get the most out of your question bank system? Here are our top five tips.

Let’s face facts – not all questions perform as well as you like during examinations. It takes significant amounts of time to prepare, collate and review examination questions. And finding out after your students have sat the exam that your questions have not performed as expected is disappointing – and can call the validity of your exam into question.

There can be many reasons why a question doesn’t perform as you would like. The beauty of a sophisticated question bank system is that you can use your past experience to continuously improve – easily. It is similar to the sentiment of Lord Kelvin’s comment – “if you cannot measure it, you cannot improve it.”

1.    If your question bank database is effectively a collection of word documents stored in individual files it is time to consider another solution.

This kind of approach was probably ok twenty years ago, but ultimately now it is likely that you have a stack of data that is unsearchable and un-index-able. It’s likely that this is holding you back from improving your questions from exam to exam because you can’t access powerful insights at your fingertips.

2.    Use the statistics.

It might seem obvious but really take notice of what the statistics in your question bank tell you. A sophisticated question bank system will analyse several parameters for each question or group of questions allowing you to gain an in-depth understanding of question effectiveness as well as making comparisons with its performance in previous exams.

A robust question bank system will present the following statistics at a minimum:

  • Mean – The mean score is one of the most basic stat tools. A high mean score suggests the question was relatively easy and makes distinguishing between A-level and B-level students difficult.
  • Facility – As the name implies, will give you an indication of how easy or difficult a question was. The higher the facility, the easier the question.
  • Discrimination – Is a measure of how well a question discriminates between good and poor candidates. This can be calculated in various ways, but typically it compares the top third of students with the bottom third. The higher the value, the better the correlation between the question and the exam. The quality of distractors can influence student performance on an exam question. A strong distractor will test the top scoring students. For distractors to work they must be plausible; could possibly be argued as correct; and have a reasonable chance of being selected by someone who has not learnt the material.
  • Point Biserial –  Measures the correlation between a candidate’s performance on a question and the exam as a whole. Results range from –1 to 1. A higher value indicates that the students who had high test scores got the question correct whereas students who had low test scores got the question wrong. Questions with discrimination values near or less than zero should be removed from the exam.
  • Pentile Histogram – This shows the percentage of students in each 20% score band who have answered the question correctly. An indication of a good performing question has a pentile showing a downward trend.
  • Kuder Richardson Formula 20 (KR20) value is used to give an overall indication of reliability.

3.    Use the statistics.

Yes, we have already mentioned this, but it is worthwhile considering the statistics behind OSCE (practical) exams as well as those for multiple choice exams. The set up for OSCE exams can be extremely complicated – and with multiple assessors making it even more challenging. However there are real benefits in understanding the differences between assessors and their performance during the day – especially when it comes to any subjective marking of a candidates performance.

4.    Numbering controls that are in-built in question bank system prevent simple numbering mistakes that can render your exam invalid.

Automatic controls ensure that the exam questions remain in sequence following the addition, withdrawal or movement of questions within an exam. Our recommendation is to always manage all your exam editing and re-ordering within the question bank. Occasionally we see clients creating their exam in question bank software, and then exporting the questions to another format (most likely word). It is at this point that exam questions are edited and then re-imported into the question bank system.  This process usually overrides the automatic controls that help to minimise human error – and of course introduces a numbering error.

However, the best solution for removing human error in exam preparation and management are next-generation exam management tools, such as the eSystem. eSystem manages the entire preparation, administration and analysis of exams managed within one solution. There is no need to export exams and circulate them to stakeholders as each author can review specific questions or the entire exam in a controlled and audited manner.

5.    And finally use your question bank to provide a framework for preparing your exam.

This is known as blueprinting (view this short video). When a question bank includes information on how many times a question has been used, when it was last used, how it performed and so on, you can use it as a starting point for designing new exams. When you are close to finishing the exam design phase, use your question bank to cross check that you have covered all your learning outcomes.

The power of a robust question bank system isn’t in doubt. It saves time, energy, resources – and at times stress. As well as ensuring that you set the exam that best evaluates candidates’ knowledge. Our specialists here at Speedwell are happy to give you a host of other tips which will ensure you get the most out of your question bank system. Please get in touch and ask!