Are standardized exams more reliable?

In the past standardized tests, such as CLEP, were often subject of criticism on this forum. However, it seems, we almost forgot to discuss the problems of "traditional" testing methods, involving a human examiner. In the last months there have been reports in the German media about the sociology professor Dr. Wolff-Dietrich Webler from the university of Bielefeld who is investigating the examination practice at German universities. In short, his conclusion is:
To some degree exams are a gamble.

Here are some observations he has made:
- Examination grades are more favourable later in the day because the examiners get tired.
- The mood of the examiner interferes with the grading process
- Examiners rate the same performance differently; A "C" from one examiner is worth a "B" from another.
- The knowledge about the past performance of the examenee by the examiner has an influence on the grading of the examenee.
- Questions asked during the examination do have a different level of difficulty. So, some candidates get away with "easy" assignments while others have to tackle more difficult stuff.

I think this problems are not unique to the German educational system. Can you second that? Has someone made a similar experience?

Dennis

 

I dont know what were the issues that were discussed earlier regarding CLEP testing. but yes you are right those issues with manual markings still are there.

So are we saying that CLEP (computer based) making is somewhat better or inferior ?

 

Yes Dennis, I second to that...in the Philippines though.

My instructors in engineering (during my time) never gave a student a high mark on the course they taught. Their reason was (according to them) they also had the same treatment from their instructors in the same institution during their school days. Although most of them (my instructors) were Board Placers (top 10 of the Engineering Licensure Exam).

 

I seem to remember that I once read some research article related to the fact that college entrance exams (SATs) actually had only moderate-poor predictive value. That is, there were lots of people who scored high that did poorly in college and lots of people who scored low that did well in college. The test showed predictive value only for people who scored in the middle. They showed a strong tendency to perform moderately in college. I'd be interested to hear if my memory of this is accurate and if there are any more recent findings.
Jack

 

Dennis wrote:

> To some degree exams are a gamble.

Students have the right to appeal exam grades, and if they've been wronged, they should do that.

> - Examination grades are more favourable later in the day
> because the examiners get tired.

There are ways to minimize unfairness due to this. Instead of grading each exam from beginning to end, the professor can grade everyone's answer to Question 1, then everyone's answer to Question 2, etc.

> - The mood of the examiner interferes with the grading
> process

My father usually listened to music while grading.

> - Examiners rate the same performance differently; A "C"
> from one examiner is worth a "B" from another.

When different professors are teaching different sections of the same course, they should divvy up the grading so that every professor grades some of the answers of every student.

> - The knowledge about the past performance of the
> examinee by the examiner has an influence on the grading of
> the examinee.

Only a problem for small classes. For a large class, how's the prof going to remember past performance?

> - Questions asked during the examination do have a
> different level of difficulty. So, some candidates get away
> with "easy" assignments while others have to tackle more
> difficult stuff.

Not all students find the same questions easy. Some questions will be easier on average, which may be the motive for grading students "on a curve".

The grades my father gave tended to graph not as a bell-shaped curve, but as a "two-humped camel". The department chairman would be troubled by this, and would ask my father why they weren't a bell-shaped curve. My father would reply: "It's a new concept in education: competence and incompetence."

 

You can say that about any exam. If the material you revised doesn't show up, or shows up in a form that you can't make adequate use of, it makes little difference whether you're taking a three hour written exam or a standardised test: you're still up a creek without a paddle.

Every university level exam I have taken has been double marked. In the UK, exam papers are also subject to departmental and external moderation.

When I'm one of the examiners, if I know the order the other examiner has marked the papers in, I'll either reverse it or randomise the papers.

Um. Not really. Although few things irritate me more when my own lecture notes are regurgitated back at me on an exam paper without any sign of an attempt to understand what I said, the student will be getting a bad mark because they've failed to reflect upon the question, not because they've hacked me off.

That's why double marking, moderation and the external examiner exist. The examiners meeting is usually the place for a frank and free exchange of ideas about what marks will be given, and if the two people doing the double marking can't finally agree on a grade, the examinee's paper will go to a third marker and will automatically be referred for moderation.

This one only really applies in very small classes where everybody is known personally to both examiners, a very small minority of cases. I'd also expect the moderator and the external to pick up any cases where the mark didn't reflect the performance of the candidate.

This one is also related to the question of being able to tell who somebody is despite their being identified only by an examination number. One of the few things I liked about the GMAT was that the analytical writing component was done on a computer: no chance of anybody taking one look at the handwriting on script number 80612 and going "Oh, you poor sod, you've got to try and interpret Angela's amazing seventeenth century-influenced scripthand". I consider the use of word-processing facilities (on suitably wiped machines, of course) as being one of the great leaps forward in the written exam.

Exam questions are chosen from a pool of possible questions so that they are all of similar difficulty. At the least the moderators will see the questions first, and will be able to check on them.

Also. Define "easy". I suspect I'm not the only member of my year to still have nightmares about the morning of the first Saturday in June 1985, when the "History of Europe 1500 - 1700" paper from hell left the whole of the University of Warwick's history class of 1985 in a state of near hysteria. I am. however, one of the few who got out of that exam with an excellent mark after avoiding the question that looked easy, but which everybody answered wrongly. One person's easy is another person's nightmare, depending upon taste, temperament and what one has revised.

Written exams are only as good as the system you have in place for double-checking them, which is why the UK higher educational system has invested a great deal of time, money and effort in developing the system of double marking, moderation and external examination that exists today. Add in the use of computers so that the handwriting element is removed, and it'll be even better.

The fact remains that standardised tests can't replace exams in the arts and social sciences. I'm firmly on Gavin Kennedy's side of the fence here: the only way you can test whether a candidate has understood a qualitative subject is to give them a three hour closed book exam and make them write their guts out.

Angela