Yet Another Thought on Rigor

I have yet another thought on rigor. Please consider this as a "blowing off a little steam exercise". I've probably overstated my case. I feel a lot better after writing it...

Imagine the following scenario - hypothetical, yet entirely possible ...

Student A enrolls in a DL bachelor's program. The program is open admission (no ACT, SAT, etc.) and doesn't employ any standard external outcome assessment measures (e.g. Major Field Test) to assess the validity of the degree program. The DL program doesn't require proctored exams as everything is on-line. After graduating with a BS and a high GPA, the same school recruits the student into an MBA program and the student enrolls. Again, no external measurement is required for admission (e.g. no GMAT/GRE) and no external measurement is taken upon the student's graduation. Again, there are no proctored exams because the school wants to make the program "accessible" to everyone. Nearly 40% of the faculty at this institution hold degrees from the same institution. Many of the others come from similar minded DL institutions.

Encouraged by earning an MBA with a high GPA, the student then enrolls in a PhD program at the same school. Again, there are no external measures required to enter the school (e.g. no GMAT/GRE) and a significant percentage of the faculty hold their academic degrees from the same school or similar minded institutions. The person's committee is populated with a number of graduates of the school in which he/she has earned all of his/her degrees. The balance of the committee come from similar minded DL schools. Upon graduation the student doesn't bother to publish the results of their research. In fact, the school is so "practitioner focused" that graduates have no concept of academic research or the venues where research is published.

The student proudly celebrates his/her graduation with a PhD. He/she applies for a teaching job at a traditional B&M school and is mortified that they aren't hired. Someone dared to question the student's academic credentials. So they go to teach at the same school they earned all three of their degrees at. They become fully engaged in educating another generation of students.

Sound hypothetical? I don't think it is, given some upcoming PhD programs that I see coming on the market.

I'm all for accessibility. Qualified students should be able to complete degree programs, even if they can't attend a traditional program.

But I have a real concern about quality in the DL world. What I see are too many DL and part-time degree programs with eroding standards. Should bachelor degree programs be open admission? Perhaps. But what about graduate programs? Does everyone have the "right" to earn a masters or a doctorate? The current crop of US DL programs I see effectively run open enrollment at the graduate level, turning down very few applicants. Whether they are for-profit or are an aggressive non-profit, money talks.

I also see an unwillingness to use inperfect yet statistically validated measures - be it the GRE/GMAT/Major Field test and other measures to see if incoming students are qualified or if outgoing students learned anything. I also see a reluctance to proctor exams.

I suspect that the already low opinion of DL could sink even lower as the market fills up with people that have doctorates from DL schools. Some of these folks may be highly qualified, while others are not. The degrees at the end of their name are the same and DL institutions haven't done much to filter qualified students from unqualified.

I hope I'm wrong on this, but I fear I am not.

Regards - Andy

 

Andy,

within this forum we have seen multiple examples of "PhD"s at highly accredited institutions who are academic frauds. Would it not be more appropriate to police the professors first before measuring the students. I am for standards, but applied equally to all areas of academics. There is no consistent method to measure programs from institution to institution. Accreditation was supposed to be a standard but it fails to provide adequate quality controls.

I like the idea of entrance exams, comps, major field tests, any outside review. However, these are not applied equally in all insititutions. I would even settle for all the institutions within a particular accreditor. If consistency of application existed I would agree that the tests have some merit in screening applicants since no school would accept them.

Now, when you face the first lawsuit because of cultural bias, or other discrimaination complaint, the test will be waived or abandoned. It is unfortunate, but I believe lawsuits have done as much to diminish academic entrance requirements as any of the hypotheticals you propose. DL or otherwise. Combine this thought with the fear of further lawsuits for not passing a student and the academic process continues to be dumb downed.

Outcomes (exit comps) not entrance exams are a better measure of the individual knowledge. But you have that pesky money thing that could blow up in your face when a person doesn't graduate after spending megabucks.

Can you elaborate on the proctored exams, i.e. closed or open book, essay, etc...

Lastly, I am impressed that you are concerned about the DL community, good hypothetical and certainly one in which the accreditors can play a part in limiting.

Thanks for the read....

Regards,

 

I think that I agree with most of your thoughts. I am interested in how you would view an undergrad degree earned solely through statistically validated (although perhaps imperfect) measures. For example, a degree earned through one of the big three.
Where would you rank such a degree as compared to other dl schools in general?
Where would you rank such a degree as compared to other dl schools specifically?

Thanks,
Tony

 

Andy

I have expressed views about academic rigour on other threads, so I may not have to repeat them here. It seems from what you hypothesise about that we agree that the 'dl'(?) schools you describe are outside the reputable academic community, and from the context you mention (GPA, GRE) you have in mind common practice in North America.

Rigour is ensurable by external appraisal of a School (as in the UK system of Royal Charter, regular audits by the Quality Assurance Agency, or QAA, External Examiners, and such like). Of course, when reputable Schools also apply 'innovations' that soften the exam regimes, like some of those you mention, their output quality declines. This is a universal campus teaching (CT) problem, exported into Distance Teaching (DT), a form of teaching often misnamed Distance Learning (DL). In short the problem you identify is not a uniquely, or even necessarily an essential, characteristic of DL.

Your solution appears to be to concentrate on the inputs into a learning system, both of students and faculty. This is the wrong way round. Quality is determined by the outputs, not guaranteed by the inputs (statistically valid or not). In DL there is no need for the rationing of entrants (as there is in CT and DT) up to a fairly high population of would be students.

If the exam regime is softened (continuous assessment with high risk of fraud, open books, assignments, participation in CT or DT cohort online sessions, and so on), statistically valid input measures are redundant, or at least not relevant. If the exam regime is toughened (the EBS model), independently invigilated (proctored) and externally, examined, no choice of questions, closed book, and so on, the output is academically, and I suspect statistically, valid. Can the candidates pass the same exam as everybody else?

In our experience (8,000 candidates at the recent June 04 final exams, for example), about half the so-called 'open entry' (no GPA, no GMAT, no first degree, about 14 per cent of our intake) fail their first exam, but, interestingly, the other half consists of candidates who do have previous entry qualifications (first or second degrees from reputable universities, many volunteering their GPAs and GMATs!). The output test of final exams reveals a hole in the statistically valid input criteria - they cannot pass our final exams.

I am happier with the measure of academic rigour of the final exam than I would be with what you suggest. Also, I am happier with the rigour of our MBA by DL than I am with the claimed or assumed rigour of many MBAs from reputable CT and DT Schools, including a few of those distinguished CT Schools in the UK and North America.

 

Professor Kennedy - I think a stong point can be made for measuring inputs, at least in the context of "taught" programs in the U.S. If you don't screen who is allowed in the program, you can end up with a class that is held back by unqualified students. Faculty do teach to those that are in the classroom. Further, at least in the U.S. most schools don't have the guts to say "sorry, you failed" to students.

I believe that the quality of outgoing students from a degree program is in part determined by the quality of incoming students. Harvard graduates extremely bright students - in large part do the fact that they are very selective on who they take in. As much as a school can teach a student in 2, 4 or more years, faculty can't be miracle workers. Students without adequate preperation create a huge drag on academic programs.

I need to re-read the Bell Curve, a book I spent some time with a while ago. I think my point about incoming quality was one of the themes expressed in the book.

Also, I support outcome assessment - on two levels. I agree that students need to pass rigorous, verifiable exams. Also, degree programs need to be assessed to demonstrate that they actually teach students.

My concern is that DL schools have widely variable quality and that they are not capable of "filtering" different levels of student performance.

Regards - Andy

 

Andy

"My concern is that DL schools have widely variable quality and that they are not capable of "filtering" different levels of student performance."

I think we can agree on that point. What I would add is that I believe any reputable academic institution is 'capable of filtering different levels of student performance'.

But where a School chooses not to do so - which you say "at least in the U.S. most schools don't have the guts to say "sorry, you failed" to students" - then, as we say in English, 'all bets are off'.

Being capable and deliberately choosing not to be, changes the game.

If a School has zero or near zero failure rate for its students, it is supicious. Students fail their exams for all kinds of reasons, not necessarily to do with their inherent abilities. Effort and its sustained application to learning, can produce a 'Bell' effect as can almost any other attribute. And students are distracted for all kinds of reasons - moods, depression, love interests, wandering attention off the syllabus, deviant behaviours, drugs, family problems and 'in the wrong course', medical reasons, and so on.

In France, young persons have the right to go to university. Many self-select and choose not to do so, but the entry intake displays wide variations in performance. Many fail to proceed each year. Some 'give up' and pursue other options; many are 'failed' in academia but move onto more relevant careers in sub-academe courses and in the job market. Students who 'know' the courses are not for them, tend to quit. Those who stick it out also are varied (the elite at the top really are an elite) and many who fail or quit could have made it through.

Hence, if everybody passes at Harvard I suspect it has less to do with the absence of a normal performance curve in the US than with the 'massage' of faculty of the results. The notion that the 'best' class produces performances of 90+% for near everybody is suspect, otherwise known as grade inflation. If you raise the grades of performance at the bottom of a class you squeeze the alleged performance of those at the top to an incredible, thereby, unbelievable, narrow band of percentage marks.

I do not think this is a DL problem particularly. It is a mallaise of the CT system. 'Afraid' to fail failures to perform is dangerous - I trust US pilot schools are more courageous!

Kind regards and thanks for the exchange of views. It has helped my clarify mine.

 

Thoughts on Rigor

I'm not sure measuring inputs are important in a mentor-learner model of which many DL programs are comprised. I'm positive they are vital in a classic instructor-unenlightened model. In this model, lecture and other non-interactive teaching mechanisms are used in addition to exams. In this case, measuring inputs would disperse individuals into knowledge gradients thus allowing similarily positioned students to move forward as a cohort. In the standard American teacher-student model (which is defined by somewhat looser organization structure than say an European schools model, in which signifigant time and effort is spent in answering questions and dealing with individualized educational issues), on the other hand, input measurement performs a secondary function. Rather than to exclude, it is used as a tool to allow the teachers better plan, deliver, and tailor education to the bulk of the studnets in a cohort.

I think that inputs are best used, therefore in determining gradiency and instruction tailoring.

In a pure DL model (of which I think the mentor-learner model is a good measure of purity), the mentor would first guide the student to learning resources and then secondly test the student on knowledge of the materials.

I'm not sure inputs can be used. Suppose we take a classic youngster fresh out of the secondary education system and walk him into this model. He takes an entrance exam and knows just enough to pass the test. He follow the prescribed path and earns his degree. That's all fine and dandy. What about a 45 year old computer programmer who never earned his first post-secondary degree, but has been performing at a satisfactory level (in his employer's mind) for 20 years. He passes the test with flying colors and then proceeds to follow the same prescribed path to earn his degree that was followed by the youngster.

My question is this: At some point basic skills such as elementary reading and computation are necessary to begin collegiate work that's true enough. But the converse of that is that for the middle-aged man there is no incentive for doing well on the entrance exam. So long as he passes, he's fine. If there is a cut-off point for dropping a potential student of any age, shouldn't there be a cut off point for granting credit based on say scoring a perfect score on the entrance exam?

I agree with the Professor. Outputs and demonstrables are the essentials. If my 45 year-old programmer can demonstrate knowledge on a collegiate level through tests or trial experience, isn't it redundant to have him take a basic skills assessment?

I personally just took a basic skills-preassessment at a fairly new US regionally-accredited facility prior to being accepted. Now, I'm not a genius but I've had close to 5 years of college-level work at different RA colleges throughout my life. I'm accustomed to tension and testing experiences. This entrance exam, while very long, and could easily be impossible for a mid to late-secondary school student to pass, was rediculusly(sp) easy for me. I'm not sure the three hour test did anything for me.

Now, on the rigor issue. What's hard for me may not be hard for you.

I'm not sure rigor is a quantitative idea. I think one might be able to approximately measure rigor by determining the number of failures verses attempts but failure won't affect a super-genius nor a very poorly prepared student. The poorly prepared student would be used to failure, while the super-genius will have never failed in his life and won't fail this time.

That's like saying beauty is a quantitative idea. One can determine a beautiful person by the number of successes verses attempts. If the number of attempts is low and the number of successes is high, then you've successfully measured beauty. I just don't think the world works like that.

 

Not so clear

A small number of points/observations:

1. The exam regime Professor Kennedy keeps on about limits access to learning and development. It suits some but not others. People learn in many different ways and all exam regimes do is allow those who are good at exams to proceed whereas we flounder about when it comes to exploring th emany other ways in which people learn and acquire skills, knowledge, etc. I think we are in the 'extinction' phase of learning/teaching as we know it. We are not being extravagant enough to explore the psychological and cultural foundations of the adult learning process. The challenge is to produce a truly international curriculum that is taught in a truly international way.

2. Regional Accreditation in the USA has never been a hallmark of academic quality! It is a hallmark for input (finance, facilities, PhD credentials checking, etc) but rarely do inspection teams, if ever, speak to students and/or look at academic work and marking to ensure academic quality. This is a major failing of the US RA system. I have recently visited an RA accredited institution and reviewed final student year examinations and grades and fewer than 25% should have received a degree in my view!

3. The more we allow (as is about to happen in the UK) more institutions to award degrees and be the guardians of academic standards the more we dilute them!

AJJ

 

Andy,


I teach for a B&M University and the number of students admitted in to the PhD program is about 3 or 4 out of an average of 100 applicants. We normally have 1 or 2 PhD graduates a year and yet we find that they have a very hard time getting employment in the local market since Universities normally hire one or two faculty members once in a while. Most of them hunt for jobs in the US or go to another provinces in Canada. If you look at the stats of the DL learning business institutions, they graduate business PhDs in the order of hundreds, Is this really a good practice? What are their chances of gaining employment in positions where the PhD is required? Do we really need so many PhDs? I think that the market will be flooded pretty soon with PhDs from DL instituions as they graduate 50 or more times people than traditional institutions.

 

Somehow I get the impression that Andy is fighting straw-men.

"DL" is not synonymous with "shitty". It's just a remote delivery medium.

Actually, I think that Andy's real target is expanded educational opportunity. DL provides that to people in remote locations. Flexible admissions standards and the like, whether DL or B&M, provide it to non-traditional student populations. There's the matter of preserving the anthropological rite-of-passage aspect of the doctorate and the labor conditions of the professoriate. And there's that damned profit motive, confusing things still further.

My point is that a lot of variables are being confused here, at least as I see it.

I also need to say that this stuff offends me personally.

My betters may be contemptuous of the riff-raff that DL, or continuing education, or something... lets in, but I'm very much part of that riff-raff.

I'm just a guy who loves education and who intends to continue thinking and learning until I no longer can. I'm extremely thankful for the opportunities that I've been given to do that.

It saddens me to see the contempt with which my teachers view me just because I'm a non-traditional student who presumes to learn. I enjoy higher education less and less because of that.

I sense that many professors see the paradigm of higher education in the professoriate's own reproductive cycle. Education that doesn't conform to what the professors went through themselves in becoming professors is inferior, simply by definition. Education for goals other than generating new professors is education for lesser goals, simply by definition.

This is part of the reason why my interest is shifting to the tremendous number of non-credit, non-degree opportunities out there that don't interest Degreeinfo.

Many of these provide pure learning, offered simply for its own sake. I make absolutely no claims about them and in turn I ask nothing of any of you.

 

Re: Re: Yet Another Thought on Rigor

RFValue - I have to agree. A number of schools are considering or already have added PhD or DBA programs in business. The market will be flooded soon, if it hasn't been already.

What happens when a market is flooded with "product"? I suspect that either wage rates will decline and/or good product will chase out the bad. In terms of PhDs who do you think will get jobs? DL grads? Or B&M grads?

Regards - Andy

 

Professor Kennedy - I think I understand the EBS model and I don't have a big problem with it. I'm certainly more familiar with traditional classroom teaching - but I'm sure there is a place for the EBS method. The notion of rigorous, proctored exams is very appealing.

One point of clarification - when you talk about "outcomes", I think you are speaking of course outcomes, not degree program outcomes.

The difference is quite significant. If you admit a student without doing any screening but you make the student take a proctored test after the first course - you're accomplishing the screening task in an only slightly different way than I advocate. You're measuring course outcomes and you're giving the student in-process feedback on their degree program. If you fail a percentage of these students, you're sending clear messages about what is acceptable performance.

The problem here in the US DL programs that are frequently talked about in this forum is that there is no up-front screening and little in process feedback. Remember, that most of the taught courses are taugt by adjuncts, many of whom tend to take the easy route and avoid failing students. I've seen grading stats from some of the DL programs we talk about here - and there aren't very many failures.

Once a student has completed several years of a DL doctorate program, schools are vey unlikely to tell the student to leave. They may of their own choosing, but rarely will a school boot them. Fear of lawsuits may be part of the reason for this.

I like the EBS mentality the more I think of it. Perhaps the addition of proctored, blind reviewed exams early in a DL doctoral program would be a solution. I do think, however, that the GMAT/GRE approach is also viable.

Regards - Andy

 

"Perhaps the addition of proctored, blind reviewed exams early in a DL doctoral program would be a solution."

I like this idea....might make a good paper topic...

 

Bill - there certainly is no intent to offend here. I'm in favor of exapnded opportunities for qualified students - and I activity teach students who are non-traditional.

For student who just want to learn for the sake of learning - non-credit or certificate programs are a great idea. Students can take courses on a non-degree basis as well.

I agree that DL doesn't have to be poor. In the business school world Case Western and UK/Australian universities show that.

What I'm battling is declining standards, particularly in the U.S. DL schools that we speak of most here in this NG. While there always be variation in the quality of educational programs and their graduates, don't there need to be minimum standards?

I see a rush of new DL doctoral programs coming on-line. Schools seem to see this as a "growth area". That would be ok, if quality were built in to these programs. But what I see coming is a flood of graduates, some who are very poorly qualified. If all these folks wanted was to learn for the sake of learning, this wouldn't be a problem. But I suspect that this new flood of graduates will be teachers of tomorrow's DL students. And the cycle continues...

I read a lot of attacks here on degree mills. I'm not in favor of mills, but the continued watering down of academic programs could lead to the point that there isn't much of a difference between mills and RA schools.

Perhaps I am battling a straw man, but somehow i don't think I am....

 

Re: Not so clear

Was this a specifically DL institution?

 

Most of the traditional universities do not see doctoral programs as a "growth area" but some source of potential researchers that can benefit to the university in terms of publications and teaching support. Also, most of the students conduct PhD research funded by federal goverment agencies.


As I said, most of the B&M universities that I know only accept few candidates every year for PhD student positions. On the other hand, DL schools see the PhD programs as a great opportunity for source of income since traditional B&M universities can only take few potential candidates. The problem is that in order to have a credible DL PhD, this will require full time faculty taking care of only few students and this would mean that a University would only be able to take only few students per year. This is a problem for for-profit organizations since the cost to
do this will be just too much and only few students will be willing to pay fortunes for this kind of doctorate.


I think that DL institutions are forced to low standards because they have the pressure to make profits. Most of the B&M institutions make their money with their Bachelor and Master's programs and the PhD programs are mainly used for research support since most of the students get funding from federal or goverment agencies. But if you are a DL institution where you main source is doctoral students, how can you make a credible doctoral degree and still be cheap enough to be attractive for potential students?

 

I have to agree with this. If a DL student is capable of passing an exam in the same conditions as a regular student at a traditional university, why would we think that he has achieved less than a traditional student? I think that having traditional assements in DL programs in a good way of making DL programs more credible.

Andy has talked about GMATs and GREs to filter out students but most of the UK
and Australian universities do not use them and yet their quality is not put into
question. The new DL model of participating in online discussions and submitting some weekly papers instead of writting rigorous exams is what is making DL less credible.

 

In your first post you sounded like you were ridiculing the idea of a school that sought to make its "program 'accessible' to everyone". You seemed to see that as a failing, while I see it as a strength.

You went on to ask "Does everyone have the "right" to earn a masters or a doctorate?" I see no problem with answering 'yes', so long as the degree was in fact legitimately earned.

Perhaps I'm misunderstanding you, but the difference between us seems to be profound: You appear to favor preventing people from learning, while I favor giving qualified individuals a chance to follow their dream, to at least try. Obviously admission to a graduate degree would presume the successful completion of a suitable undergraduate program and so on. But people should always have options available to improve themselves.

None of this means that elite, highly selective programs can't exist too, if people want them. Nothing is saying that competitive hiring situations can't favor graduates of those programs.

Frankly, I would strongly welcome the appearance of some more scholarly DL programs. As I've written before, I don't think that DL will have arrived, at least at the doctoral level, until people in a specialty can't stay current without reading work that comes from a DL program.

But that doesn't mean that all the rest of higher education, education that isn't intended to reproduce research scholars, needs to be swept away. That's the basis of my objection.

If you are gonna denounce 'em, you probably oughta name 'em.

Assuming that your charges are credible, I'm curious whether they are program-specific deficiencies, discipline-specific, or whether they discredit American DL in general, as you seem to imply.

In the areas of interest to me, philosophy and religion (in the religious studies as opposed to evangelical theology sense), DL doctoral programs simply don't exist. They don't exist in history. I only know of one in classics, with admission restricted to those already teaching. There's virtually nothing in the sciences or mathematics. They don't really exist in law.

The only DL doctoral programs I know of seem to be practitioner, as opposed to professor, oriented things in psychology, business and education. There aren't really a whole lot of these.

I realize that Degreeinfo is unnaturally fascinated by DL doctoral programs, but they seem to be a minute part of the DL world. They don't seem to justify trashing DL in general.

 

government statistics 2000-2001. 1180 out of over 40000 PhDs were in Business.

http://www.nces.ed.gov/programs/digest/d02/tables/dt254.asp

To some of the points presented by Andy and RFValve. There does seem to be a possibility that if PhDs in business or otherwise are more available via DL that at some point the DL graduate will be in a position to teach at at a significant level.
As for dilution as a whole, with this many degrees being conferred and the limited fields offered via DL I don't see it as a possibility in the near future.

If somebody reads this table better please correct me.


Bill,

I agree with your approach of making education available. I also agree with wanting specifics and issues so that if my program is on the list I can address it from within. I think that improvements only occur through concern and positive action.

 

I don't think Andy meant this is in this way. "Accesible to everyone" is more for those students that just pay their way in instead of earning the right to attend to a specific program.

The original DL programs given by traditional universities are "Accesible to everyone" but they don't low standards
in order to attact more students. Waterloo University had its distance programs for decades and their DL degrees are not less that those earned the traditional way. Students are still required to come to campus to write their final exams or do proctored exams at regular universities. Many army officials in Canada earn their degrees from Waterloo and yet Waterloo is classified as a top Canadian University. However, Waterloo DL programs are not as popular since students (at least the ones that have talked to me) tell me that this type of program is normally harder than the traditional one given the lack of one-to-one guidance.

Andy's "accesibility" refers to schools that low traditional standards in order to be more attractive. No proctored exams, no GMATS, no or very little math courses, accelerated times and so on. Why someone should be able to complete a DL degree faster than a traditional degree?.

As for getting Master's and PhDs is not for everyone, I totally agree with this. A master's degree should show that someone master's a specific field. Most of the new DL MBA programs only require to complete 10 basic business courses and you get a "Master of Business", how can you say that you master the business area by just completing 10 basic courses?.

I think that DL is a great opportunity for students to gain knowledge and advance in their careers, the problem is that some people have abused DL to make quick bucks.