CPC

CPC

< Back to Administration

Multiple Choice Papers – What’s Going On?

Mike Grant
Mike Grant
Posts: 142

 

... a somewhat rhetorical question as I realise that nobody apart from the examiners will be able to answer.

 

I ask it though because clearly something has changed recently.  If you look at the cumulative pass rate statistics it has always been the case that more succeed at P1 and R1 than their case study counterparts – and that is exactly what I would expect – picking an answer must be more straightforward than providing one.

 

This time though the pass rates are:

 

P1: 57%

 

P2: 74%

 

R1: 52%

 

R2: 65%

 

It reflects a pattern which began last September.

 

Why should this be?  I assume the quality of candidates and teachers has not changed significantly and the answer must lie with the questions being asked on the multiple choice papers.  I wonder:

 

- are they more obscure or badly constructed?

 

- are they mismarked?

 

- are they off-topic?

 

If I am teaching the wrong stuff, I would like to know so that adjustments can be made - but there is no machinery to provide for that.

 

I have long argued that the multiple choice question bank should be open to scrutiny.  Opponents claim that students would just learn needed answers.   I’ve no idea how many questions are banked but it would be quite a memory challenge.  I also think it an impractical task without gaining an understanding of the concepts in play – and that after all is what they need to be competent managers.

 

All modern examinations should offer a level of transparency – but P1 & R1 are stuck in the “dark ages” of secrecy and opaqueness.  Only examiners are privy to the process and know what answers are wanted.  There is no realistic option to appeal against any result.

 

I read in a recent OCR recruitment notice that by becoming an examiner they would acquire “an understanding of how assessment decisions are made which enables them to prepare their students more effectively for examinations.”  That’s an added ”skewing of the pitch” that needs to be addressed.  You shouldn’t have to be an insider to know what’s going on.

 

It now feels that the “goalposts” have been moved - but only the examining elite will know where to!

 

Replies

Mike Grant
Mike Grant
Posts: 142

Thanks for that insight, Nationwide.

I think it’s a valid argument that case study training is improving - which is good and the feedback is invaluable - but I highlighted the comparison because I am now finding that I increasingly have candidates who I consider competent, passing what I view to be the harder exam but failing multiple choice papers.

I’ve always monitored the pass rate statistics and since the change to the present format in 2012 the cumulative averages are P1 – 64% and R1- 60%. 

Figures for March 2016 are P1 – 57% and R1- 52%.  I feel sure something has changed even if candidates are unable to report on it.

One thing I did pick up on was in a recent paper.  Students needed to know for a traffic accident question if a llama would be classed as “cattle” for RTA purposes.  It was beyond me – and still is!  Are questions getting more obscure?

I spent a little bit of time a few years back working for OCR on a committee which looked at multiple choice questions after the exam.  That committee was disbanded for reasons that I never properly understood but it frequently pinpointed errors and ambiguities in questions and I wonder if such incidents might be on the rise again.

Sadly the only information we are allowed about the process are the pass rates and they seem to have taken a marked turn for the worse.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

nationwide
nationwide
Posts: 10
Quote:
Originally posted by Mike Grant, 8:31, 20th April 2016

Thanks for that insight, Nationwide.

I think it’s a valid argument that case study training is improving - which is good and the feedback is invaluable - but I highlighted the comparison because I ...

Read more
Read less

Thanks for that insight, Nationwide.

I think it’s a valid argument that case study training is improving - which is good and the feedback is invaluable - but I highlighted the comparison because I am now finding that I increasingly have candidates who I consider competent, passing what I view to be the harder exam but failing multiple choice papers.

I’ve always monitored the pass rate statistics and since the change to the present format in 2012 the cumulative averages are P1 – 64% and R1- 60%. 

Figures for March 2016 are P1 – 57% and R1- 52%.  I feel sure something has changed even if candidates are unable to report on it.

One thing I did pick up on was in a recent paper.  Students needed to know for a traffic accident question if a llama would be classed as “cattle” for RTA purposes.  It was beyond me – and still is!  Are questions getting more obscure?

I spent a little bit of time a few years back working for OCR on a committee which looked at multiple choice questions after the exam.  That committee was disbanded for reasons that I never properly understood but it frequently pinpointed errors and ambiguities in questions and I wonder if such incidents might be on the rise again.

Sadly the only information we are allowed about the process are the pass rates and they seem to have taken a marked turn for the worse.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I agree Mike, multiple choice pass rates appear to be falling. I am finding however, that despite my best endeavours and advice to candidates, many seem to think that they can 'get away' without learning facts, as its an open book exam, then when confronted with the multiple choice paper, suddenly realise that they do need to know their stuff after all !

I am convinced that this is having an effect on the relative pass rates. 

nationwide
nationwide
Posts: 10

Looks fairly obvious to me Mike, the multiple choice pass rate has not changed significantly for a long time, whereas the case study pass rate has improved over time, very significantly in this session. Suggests to me that trainers and training centres are getting used to the case study format. After all, there are now lots of case study past papers on the OCR website for candidates to practise. Can't do that with multiple choice !!

Doesn't appear to me that anything's changed, multiple choice questions appear to be much the same as always, according to what candidates tell me after the exam

 

nationwide
nationwide
Posts: 10

Looks fairly obvious to me Mike, the multiple choice pass rate has not changed significantly for a long time, whereas the case study pass rate has improved over time, very significantly in this session. Suggests to me that trainers and training centres are getting used to the case study format. After all, there are now lots of case study past papers on the OCR website for candidates to practise. Can't do that with multiple choice !!

Doesn't appear to me that anything's changed, multiple choice questions appear to be much the same as always, according to what candidates tell me after the exam