Exam regimes

I have expressed views on the need for tough exam regimes in distance learning programmes because many of the fads of softer regimes do not travel well in distance learning. I quote:

"A new plagiarism detection service free ot all British universities and colleges expects to test half a million student essays this year. It will compare them with a database of original essays, essays offered for sale on 'cheatsites' and the content of more than 800 hundred million websites. Meanwhile the department of librarianship and information management at Queen Margaret University College, Edinburgh has become the first in Britain to be publicly condemned for failing to spot plagiarism among its students. The Quality Assurance Agency's inspectors found 'strong prima facie evidence of plaigiarism and/or collusion during their scrutiny of a sample of student work from the MSc health informatics programme, which had not been detected either by the internal examiner or the external examiners.' The degree is assessed entirely by coursework. The inspectors added that they had 'serious concerns with the security of the assessment procedures practised in the department across all the information and management programmes, and its ability to maintain standards in the future.' Queen Margaret has now withdrawn its BA(Hons) in information management, which is also assessed by coursework. Might exams be the answer?"
Daily Telegraoh, 12 February 2003.

If campus institutions have these serious problems of the provenance of students' degree awarding course work (and all manifestations of so-called continuous assessment), it is evident that the problems of the provenance of DL students' course work, assignments, projects and theses, are much worse.

That some defend the use of such assessment methods despite the risk - and the evidence - of abuse is puzzling to me. Despite all its defects, there is no substitute for 3-hour, closed book exams, no choice of questions, invigilation by independent persons (proctors), graded by internal faculty and Externally Examined by senior faculty from other universities.

Hence, when I read of assessment methods for DL Masters' degrees that do not have tough but fair exam regimes, I am inclined to comment that these softer regimes, 'say it all'.

 

I think Professor Kennedy makes a very good point.

I think plagiarism and collusion are probably more rampant in residential programs than in DL programs -- because the opportunities to copy or cheat are more readily apparent on campus. But on the other hand, stories about professional cheating rings in some countries do give one pause. In addition, many DL programs avoid proctored exams because of the inconvenience. Personally, I think these programs are worthless (pure research degrees excepted).

Of course, the rigors of Professor Kennedy's exam protocol (independent invigilation, external examiners, etc) are met by no U.S. program. One UK DL program that I pursued was in some ways even more rigorous than Heriot-Watt's because the external examiner would select students to "viva" -- and he/she would then get the candidate on the phone for a verbal question and answer session. We had to agree at the start of the program to make ourselves available for these "viva" sessions as required.

 

Oxpecker: "...he/she would then get the candidate on the phone for a verbal question and answer session."

John: And how would the examiner at the school know that you didn't have your smart brother standing there to do the viva for you?

 

My brother is as stupid as a chicken. But I guess if he were to pursue the program then he should probably ask me to stand in for him.

I'm not sure about how the "viva" is set up, having not been "viva'ed" myself. My previous post made it appear as though the external just picked up the phone and called, but actually it's an arranged conference call with the external examiner plus the internal examiner(s). Perhaps they set it up at the examination center?

 

Dead horses

I am surprised that the issues I raised in this thread have received so little comment in a forum dedicated to distance learning. A deafening silence seems an apt description of the reaction, so far.

Then I realised something I should have noticed from earlier 'debates ' we have had about exam rigour (some rather robust). This site is less about providers of distance learning and more about users (mainly students). Hence, messages pointing out the fallacy of soft exam regimes, options for cheating, and the eventual denigration of distance learning as a suspect route to a degree directly challenge those who stand to benefit from leaving things as they stand, as long as they graduate before the curtain comes down.

I have noticed - even commented once - on the number of people looking here for the cheapest (possibly OK from poor countries but not, surely, the richest country in the world), easiest (never a good option) and fastest (always suspect) route to a degree award. I did not connect these motives with the bulk of those who are silent on the issues of fraud and cheating.

It is always a waste of time to flog a dead horse. Unless providers raise and maintain high standards of examination in DL it will remain a denigrated option and the campus Neanderthals will win. This does not stop some agencies earning a living pandering to the very forces that undermine the drive for high standards. EBS is not going to join them.

 

Re: Dead horses

I would guess that the majority of people do agree with you, but have nothing of substance to add. Some of the same people who have argued that the validity of higher education is declining are the same who complain when one comments on something outside of their expertise. Exam regimes are outside my field of knowledge. I read your post, thought about it a little bit, looked up a few different schools and compared their setup, thought about your post a little more, and went on with my browsing.

Tony

For what it's worth, I see your point, and it could hamper DL from gaining more acceptance or even losing some of the previously earned acceptance.

 

GO EBS!

Professor Kennedy,

Perhaps this is not such a contentious issue. I participate in this forum as a consumer of DL and I place great value on examinations. However, I think one can have a rigorous program by utilizing examinations as just one of the factors in assessing a final grade.

Is administering one three-hour exam the only proper way of assessing a graduate level business course?

The MBA that I am working on at Liberty University is assessed by examinations and writing assignments. The examinations are supervised by a proctor approved by the university.

The EBS MBA looks like a super program. I have no doubt that, after preparation, I would be quite successful writing EBS exams. I specifically did not consider EBS, when researching MBA programs, because EBS does not meet my tuition assistance requirements.

 

Least worst

Explanations are accepted.

Perhaps I was over influenced by memories of an article written by Harry Johnston (Chicago and LSE) on the economic conflict of interests between faculty and students - the latter always trying to lower the pass criteria (join the labour market) and the former trying to raise them (job protection).

Charles asks: "Is administering one three-hour exam the only proper way of assessing a graduate level business course?"

The answer, in the context of a distance learning degree, is ''Yes'. The scope for fraud in 'assignments' that count towards a pass is a serious obstacle to grading by other than exams. Even on campus, where there is recognisable contact between faculty, administration and students, the 'assignments' (in whatever guise they appear) are subject to fraud and the incidence of exposed fraud is increasing. This discredits the degree awards and undermines ocnfidence in the format.

In distance learning, already a format that is questioned, publicity of exposed fraud serious undermines the DL format. In these circumstances the choice is between a rigorous exam regime that minimises fraud, especially if the invigilators are independent (and not chosen by the student), to levels common on campus, and between trying to replicate examination processes found on campus courses (though, for reasons mentioned open to fraud) in a dl format (inviting rampant fraud). For these reasons we chose the 3-hour closed book, no choice of questions, etc., approach.

If it is not appropriate to do so, then dl is unworkable (because vulnerable to being discredited) in a quality assured environment. This has less to do with educational issues than with the provenance that the person whose name is on the degree certificate is the one that was examined by the system.

The dl community must address this issue.

 

The Right to Cheat?!

In an article, A bird's eyeview - The menace ahead , appearing in The News on Sunday, one N.A. Bhatti has this to say:

Opherus

 

Re: Least worst

You seem to be saying that you cannot conceive of a better method of assessment, and therefore one cannot exist. Where tenacity rules, progress is necessarily thwarted. Perhaps this mindset, and a lack of desire by forum members to debate a stone wall, is the real reason you did not find whatever response you were seeking.

 

Point scoring

Gary

If this is about scoring points and not the awareness of the seriousness of the issues for distance learning you have hit a hatrick. Meanwhile, administering a distance learning programme requires I pass on the points and stick with the issues.

When asked a direct question it is incumbent on me to answer it. You do not need to do so, but the question still remains. If cheating is evident on campus examination processes, and DL is vulnerable to a higher degree to these cheating incidents, what exam regime for DL minimises cheating (at least down to campus levels) and which set of examination processes keep it vulnerable to a higher degree?

This is not about theoretical methods of assessment. We are not in the classroom. This is the reality in the real world. Why you call me intransigent - other than for a debating floruish - I do not know. If I give a definite answer to a definite question it is because I have been involved in this debate for 12 years as a practitioner in DL degree programmes, and asa campus academic for 32.

If you wonder if I am asserting that I cannot conceive of a 'better system' that the current ones that are vulnerable to cheating on a mass scale, then I am bound to say I cannot, certainly not within the financial and administrative and intrusive boundaries that would be unacceptable to either candidates, or the institutions that finance higher education.

If you can conceive of something for the DL context (my original statement) that is practical and affordable then I am wide open to considering it. 'Good reason by force must give way to beter' (Shakespeare). Point scoring is not 'good reason'.

 

DBA COURSE ASSIGNMENTS?

Professor Kennedy,

Does EBS plan on invigilating its DBA course assignments?

Could 30% of a mark be vulnerable to cheating?


"How is the DBA assessed?

Each subject is assessed by 1 examination lasting 3 hours. The general format is 4 answers to 4 questions (no choice). Some subjects also have an assignment. In such cases the examination contributes 70% of the overall mark while the assignment contributes 30%."

http://cws.ebsmba.com/dba/dba_main3.htm

 

No change to DL EBS exam regime

I am glad you asked this question as it was the subject of a discussion at today's management meeting where I confirmed that the EBS exam regime is by final examination only.

While there is provision in the regulations for asessment by examination it is not used in almost every case, except in a tiny minority of cases where an on campus student makes a case and we are sure of the enforcement of the protective procedures for assuring provenance.

The taught part of the DBA is by exam only (i.e., for the MSc in Strategic Focus). The thesis part of the DBA is of coruse examined under the normal rules of the University for doctoral theses, plus the usual viva voce.

For the reasons rehearsed above. Documentation saying to the contrary is withdrawn not only on grounds of fact but also on grounds that it was not passed across my desk.

 

different perspectives, students vs. faculty/administration

I think the central problem here (which has been alluded to in previous posts) is that students and faculty/administration generally come at this issue from different directions. I would venture that most HONEST students would argue that a three-hour closed book, unseen exam is perhaps not the best way to test their knowledge. To be assessed entirely by written work that is completed during the course of the class is a perfectly good way to assess work, as long as the work is done honestly. Of course, that’s the rub.

In my undergraduate days, I had a math professor who weighted his weekly problem sets at something like 80% of the final grade, with the midterm and final comprising the other 20%. He gave extremely challenging problem sets, and extremely easy exams. His rational was the following. Your grade should be based on what you spend the most time on. Spending somewhere between 6-20 hours per week for each problem set, times 10 weeks, means that the problem sets take up 60-200 hours of time – versus perhaps 6 hours of exam time. Again, for the honest student, putting the grading emphasis on work that might take up to 200 hours of time seems a totally appropriate way to have one’s work assessed. Predictably, though, there were many students who earned high grades by merely copying the problem sets of others, and so for them this was a totally inappropriate way to be assessed.

For me, having most of my grade depend on 3 hours of time in an exam room makes me nervous. There is always the possibility that a poorly conceived exam will not do a very good job of teasing out one’s knowledge. Any professor can devise an exam that no student can do, or at least do well. And , invariably, most students will learn far more during the course of their studies than they will ever be able to show during a three-hour exam.

Nevertheless, I agree completely with Professor Kennedy. Because it’s inevitable that there will be a few who abuse the system, distance learning courses must, I believe, put a heavy emphasis on the kind of testing that Professor Kennedy prescribes. It’s the price we pay for the convenience of the distance-learning format. Otherwise we run the risk of a few shirkers ruining it for the rest of us. Also, I believe that most DL programs that subscribe to this rigorous testing format (HW and UoL, for example), do a very good job of devising fair, intelligent, and carefully constructed exams. Yes, given my druthers, I would prefer it if Professor Kennedy and other potential evaluators could just fly to my home and assess my knowledge by initiating long discussions into the night over tea or coffee. Absent that, though, I agree that the three-hour exam should be the standard. Somehow it seems akin to the idea in the US justice system that it is better to let a few guilty men go free than convict an innocent man. You have to give up a little something to keep it fair for everyone.

Tom57

 

19th century Oxford is not scalable

Hi Tom57

I appreciate your points. I am however stuck with the DL problem. To be scalable, economically as well as practically, the idea of faculty visiting several thousand students in over 100 countries is just not feasible.

Most DL programmes seem to cater for less than 200 people - some for many fewer - and the demand for education exceeds by wide margins even every university in the world gearing up for only 200 DL students. It is unlikely that it could be done for campus students, hence, campus PhD programmes tend to be confined to under 20 students.

It is not about what is the best way to assess students. The old Oxford type system of individual one-on-one tutorials cannot be replicated for modern conditions - a few dozen in each Oxford College, no women students, and a leisurely long summer vacation. The provenance of the term work was not in doubt and the decisions of faculty were final (exam papers burnt after each final exam to avoid appeals). After three years they knew their students abilities intimately. OK when less than 3 per cent of the educable population went to university - somewhat questionable when 40 to 50 per cent now go (actually a nonsense of the meaning of 'educable', but that's another story).

For DL final exams only - no assessment of so-called course work - is the only answer when hundreds and thousands sit them. If this is compromised, DL will collapse along with standards, against which grade inflation on campus will appear marginal when nobody trusts any candidate's performance.

I am open to suggestions, as always (please no abuse, as I am as dedicated to opening opportunities to higher education for all as anybody is).