Proposal for a blindfold test of accredited vs. unaccredited dissertations

In response to a post relating to the quality of unaccredited schools on the DegreeInformation forum, I posted the following response -- and it seems entirely appropriate to post it here as well. Here goes:
--------------------------------------------------------------
I have been writing, for more than 20 years, that the quality of a degree -- not the usefulness, necessarily, but the quality -- should be judged primarily on (1) the work that was done to earn it, and (2) the qualifications of the people who evaluated it.

I have suggested many times in recent years a Turing-type Test, in which (for instance) 5 Ph.D. dissertations are randomly selected from the shelves of a given unaccredited school, and 5 more from the shelves of a regionally-accredited school. Identifying marks are removed. The ten dissertations are submitted for evaluation to a group of scholars in the field of study, agreed to by both schools. In one protocol, the group of scholars would not be told of the nature of the experiment, and would be asked to evaluate the ten dissertations vis a vis their standards for a dissertation: unsatisfactory, barely satisfactory, and so on. In another protocol, they would be told the nature of the experiment and would be asked to divide the ten into two piles of five each.

It seems to me that the unaccredited schools would have little or nothing to lose by participating in such an experiment. If it went one way, then my colleagues, whom many here scorn, would say, "Ha ha ha, we knew it." And if it went the other way, what a wonderful marketing and arguing tool it would be for the unaccredited school, and those who support it.

I proposed this to my colleagues at Greenwich University in 1991. There was definite interest, but soon after that, I departed, and then Greenwich set sail for the South Pacific, and, well, it just never happened.

So how about doing it now? Since I do not plan to write any more books about higher education, or to be involved with any school, ever again, my interest is somewhat diminished, but still fairly high, out of sheer fascination.

Thoughts on protocol.

I think it is important that the dissertations be randomly chosen, to the extent possible. Even the worst school might have some good ones, and even the best school might have some bad ones.

For the RA school, I would envision producing a list of all the dissertations in a given field, then selecting five using a table of random numbers or equivalent, and purchasing those from Dissertation Abstracts.

For the unaccredited schools, I'm not sure. I've been in such schools where there were dissertations shelved in the library (e.g., California Coast, Pacfic Western, Kensington, Columbia Pacific, Greenwich). In such a situation, I could envision an impartial person or group going in with a table of random numbers, and selecting works from the shelves, counting from the right or the left, etc. For schools that don't have an easily-visitable facility . . . well, perhaps some of the alumni here could offer some thoughts on how that could be handled.

I would be glad to contribute both some time and some money to the project, but I most certainly don't want to be the sole participant or donor.

Responses will be welcome, either here or privately ([email protected]). And I shall now go and post this message at DegreeInfo.com, to see what sort of response it produces over there.

John

 

Is this only for English speaking or any aother nation and languige?

Is the study restricted only to US?

Learner

 

This is an excellent idea. Why not use five each? In order words, why not increase it to five dissertations from traditional schools, five from DL schools, and five from unaccredited but real schools?

 

This would certainly be an interesting experiment. My only concern is that the category "unaccredited schools" includes a wide variety of institutions ranging from "very good" to "blatant degree mill." I would have some concern that should the unaccredited school do poorly in this experiment then all other unaccredited schools would say, "Oh, well that school is obviously not as good as our school," or in the case where the unaccredited school does rather well, the others would say, "See, they're great and so we must be as well." In other words, the results would be either under or over-generalized. Care would need to be taken in drawing conclusions. Beyond that I like the idea. I'd also like to nominate Knightsbridge as the first unaccredited school to participate. We've been told (IIRC) that they do have a library that includes their collection of dissertations.
Jack

 

Good idea. It will never convince special pleaders, but it's a good idea.

 

It would be very interesting.

I would like to see each expert rank all ten dissertations plus have them draw a line in the their ranked list that said below this line the quality is not up to acceptable minimum standards.

 

Thesis

Dear Dr John,

Hello, this is my first post. The International College of Christian Therapy located in Texas is unaccredited, but their course load is heavy. I personally completed two thesis research projects for Dr Gary Barkman hard work received good marks. As a member of professional associations the education committee awarded similar marks for the ICCT thesis. Yes, I am sixty years old and care about my academic work.Thank you for your time.

Sincerely,
Stae.

 

Welcome Stae,

I see you are an Aussie - where do you hail from?

Cheers,

George

 

Dr. Bear, in the past, you and I have privately discussed how interesting it would be to conduct such an experiment; here, for the sake of discussion, are a few observations. Please forgive me if I happen to lapse into the role of Devil’s Advocate.

When we speak of the proposed experiment judging the “quality of a degree,” is that valid only for a specific individual award, or does it reflect on the quality of the institution awarding the credential and therefore is applicable to most, if not all of the degrees awarded by a specific institution or types (accredited, unaccredited, etc.) of institutions? In other words, from the point of view of statistical inference, about which population will we be able to infer (if indeed we can) something from the samples?

As described, this experiment would, more than likely, only compare and allow conclusions to be made concerning two specific schools. Perhaps not even that, as 5 dissertations from each school may or may not be sufficient, considering that the correct sample size depends on the confidence level desired, the tolerable margin of error, and the variability of the populations being studied. However, I am sure that Dr. Douglas can speak to this much more eloquently than I can.

Why do I get the funny feeling that enlisting the cooperation of unaccredited schools for this project will be a difficult, if not impossible, task?

Methinks this is where it starts to get tricky…

This method would ensure a random sample, which, of course, is absolutely essential.

One of the problems with designing such an experiment, is that it must directly address some of the reasons for the experiment in the first place, such as the fact that, while information on accredited institutions is fairly public and readily available, most of the information concerning unaccredited institutions, by virtue of the fact that they have no external validation or oversight, comes directly from the institution itself. Similarly, whereas selecting a random sample from any accredited institution (or all of them collectively) is a simple and straightforward matter, the selection of a sample from an unaccredited institution depends entirely on the cooperation of, and information provided by, the institution being assessed. As the participation and cooperation of the unaccredited school(s) in question must be enlisted in advance, one must ensure that the population had not been tampered with prior to sample selection and that it actually includes all objects of interest.

Another major reason for the need of such an experiment is the uncertainty and lack of transparency concerning the process by which degrees are awarded at many unaccredited schools. Therefore, unless an unaccredited schools, in awarding degrees, claims to employ a similar process to that of accredited schools, we open the door to claims of bias as, in a sense, the dissertations from accredited schools have already gone through a process akin to “the experiment.”

In other words, there have to be sufficient safeguards in place to ensure that integrity of the data and validity of the experiment.

Although I would relish the opportunity to participate in such a project, I am concerned that, should the results prove to be unfavorable to unaccredited schools (or a specific unaccredited school), my participation would be seized upon as one of the excuses for such a result. In fact, perhaps I’ve said too much already...

 

How does that differ from somebody simply endorsing somebody else's work?

I'm inclined to think that the Google test already tells us that stuff, and probably more reliably. If a scholarly institution (whether it grants degrees or not) is producing good work, we would expect universities, researchers and professional organizations to take some kind of notice.

Any impact that a test like this would likely have is just a function of the personal authority of the judges. Maybe their clout might be best employed not as a marketing gimmick, but to open doors to getting additional attention for any valuable work that might conceivably be just gathering dust out there.

Then we could look to see what the broader community's response is.

 

Re: Re: Proposal for a blindfold test of accredited vs. unaccredited dissertations

 

There might be a maturation threat to validity of such a non-experimental study. Perhaps you would need to know the GRE scores of the authors of the dissertations to develop an understanding of whether the accredited schools are developing more proficient researchers than unaccredited schools or simply attracting better students. In other words, the accredited schools might attract the better students who would produce higher quality scholarship regardless of whether the doctoral research process took place in an accredited or unaccredited context. The case might be that unaccredited schools might be producing research of similar impact with vastly inferior students.

Best wishes,

Dave

 

I think it is a great idea. I would add that NA schools to the mix as well. At the risk of growing the study beyond your original intent, I would like to see the same experiment at the undergraduate and graduate levels.

To be a little more fair in the experiment, I don't think 5 papers can be selected from one school and 5 from another. I just don't think you'd be working with sufficient data. How do you select the 1 RA school and 1 NA school? It's just too narrow. I think to have a more representative sample you'd be better off taking 3 papers from each of 3 unaccredited, RA and NA schools (total of 9 papers), and then comparing them.


Pug

 

Re: Re: Proposal for a blindfold test of accredited vs. unaccredited dissertations

I think that you've put your finger on the crux of the probem. Most unaccredited institutions are run by dishonest thieves, IMHO. There are some reputable ones, to be sure. How could one reliable tell the difference. For example say that an institution got decent quality out of 50%. They could restrict the population to be chosen from to the good half. This 50% could be 80% bogus, or even 20%. It would be very difficult to draw reasonable conclusions in that kind of situation.

 

Dr. Bear:

Question #1: (sampling frame):

Are we talking about DL-only schools (maybe DETC or ACICS accredited vs. unaccredited entities?) or DL-and-B&M mix (DETC and/or RA accredited vs. unaccredited entities)?

Great proposal!

Questions aside, kudos to you for this Herculean bit of statistical and research design (not to mention philosophical) mettle, curiosity and ingenuity! This would definitely add to the growing body of knowledge about degree quality and/or mode of receiving such, and/or if such mode-to-quality relationship is relevant or not.

This could, presumptively and probably, be a very interesting and grand research and dissertation topic for some legitimate soon-to-be PhD candidate (or EdD for that matter). A quantitative/analytical/statistics/research design/data analyst aficionado would love to tackle this interesting experiment, if:

(a) Education (or the academic considerations and research thereof, per se) is one's field of study or practice, and if

(b) One has not already chosen a topic and is not already engrossed in one's own dissertation research efforts.

My observations on the discussions thus far:

I agree. This is an important point. However, statistically speaking, as long as the proposed research regimen adheres to the gold-standard for experimental research design - the double-blind research design, that is, selectors "blind" to one another's existence or affiliations, and selected dissertations blindly or randomly chosen) - the problem would, at least qualitatively, be taken care of.

I agree. This further buttresses the point made above by Jack Tracey, that deliberate measures be undertaken to ensure that the category "unaccredited schools" includes a wide variety of institutions ranging from "very good" to "blatant degree mill."

Excellent idea! (see below).

I agree. This point is a major one!

I have read elsewhere, for example, that Kennedy-Western "U", that paragon of long-time, supremely unaccredited entities, has posted or used to post on its "The Pub" website only for student access, a collection or sample of some of their "best" (though unaccredited) BA/BS papers, MA/MS thesis, and PhD dissertations.

It has also been reported by some, including current and former students of KWE on an alternative forum made up of current and former KW enrollees, that some of these so-called "best" KW dissertations were of fairly decent quality while others were apparently so poor in scholarship and writing that that there were even typos on the cover pages (not caught and certainly not corrected by the writer or faculty reviewer) of some of these dissertations!

Now, for argument's sake:

If the owners/administrators of, say, KW"U" were invited to participate in this study and were informed of the fact that a selection would be made from their dissertation collection, one would suspect that, since KW"U" managers know of and have seen written complaints, expressed by some students via the KW student-only "The Pub," about the poor quality of some of the posted dissertations, that they (the KW"U" managers/owners) would take steps to remove said "poor-"quality dissertations from the collection to be made available to the "blind" sample selection group or researcher(s).

One need not be a clairvoyant to deduce that managers/owners of unaccredited entities in general or KW in particular, would do this. It is the bane of human nature to protect one's "face". They probably would see it in their best interests to put their best foot forward.

Thus, Jack Tracey's point above, seconded by Gus Sainz, that random selection and inclusion of a broad array of the panoply that constitutes unaccredited dissertations - is essential for the success of this experiment. I cannot agree more!

 

To get around the tremendous amount of variance in the quality of unaccredited schools, I think you would choose one (or two or three) to pick dissertations from.

No need to limit it to 5, except in terms of workload on the evaluators. But unlike quantitative studies, I don't think the sample size is a huge determinant. After all, while we accept as an axiom that some lousy dissertations get accepted by accredited schools (mine, I've heard!), it would be unusual to find multiple examples in a selection of 5, unless the school lacked quality.

Getting at the dissertations of an unaccredited school is a problem. An even bigger problem is the opportunity to sample from their entire population of dissertations. How do we know we're getting to pick from them all, as opposed to a screened-and-selected sub-set? (CCU's dissertations-under-glass, for example.)

My bottom line: While the results of this might be interesting, it is unnecessary anymore.

The whole basis for this is to establish the legitimacy of some unaccredited school or schools--that they're putting out a comparable quality of graduate. Fine, but does this need to be established anymore? Looking back over the past 20 years, most of the schools that would be candidates for such an experiment have either (a) become accredited, (b) dropped their doctoral programs, or (c) gone out of business. In 1980, I would have chosen from Sarasota (now Argosy), Walden, Columbia Pacific, California Western (now CCU), Clayton, and perhaps one or two others. (Like Heed.) But it isn't 1980, and things have changed.

Which schools would be candidates for this today? CCU? Doctoral programs stopped. CPU? Out of business. Clayton? Selling degrees in Asia. Walden? Accredited. And so forth. These days, we're left with "the usual suspects," Kennedy-Western, SCUPS, Century, etc. Or the obvious degree mills, who wouldn't participate anyway.

Other than a few California-based niche schools, and some others (like the University of Fairfax), who would we include? Pacific Western?

No, I think this is based on an idea--the legitimate unaccredited school as a real alternative because such programs can't get accredited--that is passé.

Oh, and one other thing. The outcome (dissertation) isn't the only important measure. The process, and how the learner changes and grows, is important, too. I'm not sure that's measured in the dissertation alone, if at all.

 

First, Knightsbridge University, as its Vice-Chancellor, Henrik Fysrt Kristensen, has frequently pointed out, does not catalog or maintain a list of the dissertations in its “library.” Therefore, any list provided for this experiment would be self-selected and highly suspect.

Second, it appears that Knightsbridge University’s campus is nothing more than Henrik’s house. Guess which room in my house is designated “The Library.”

Third, since the Ministry of Science, Technology and Innovation (which oversees all higher education institutions) stated conclusively that Knightsbridge University has “no formal or legal authority to award degrees” in Denmark, it appears that Henrik and most of Knightsbridge’s vocal supporters have ceased their marketing efforts on various distance education fora. It seems that despite their persistent efforts, they were eventually sucked into that big black hole of illegitimacy.

 

Former part: arguably true; Latter (parenthetical) part: not true at all.

Dr. Douglas, do not even go there! Please do not give the dumb shills over on those other forums any ammunition for them to gloat over or flail about. They lurk and/or post here, too. You know as well as I do that sarcasm and cutting-edge wit do not go over too well with that crowd. They mostly live and breathe unsophisticated literalisms - literally! LOL!

I do not think so only because I am prone to think that scholars think and deliberate with support from empirical data while others may use in their talk the popular though unsupportable "I think," "I believe," "I expect," "perhaps," "maybe," etc.

At the very least the proposed experiment will provide some data to support what some or many of us already know or hypothesize.

Absolutely, completely, profoundly and unreservedly true!

Thanks.

 

The problem I have with the "Google test" is that unaccredited program dissertations don't get listed at UMI.

UMI is often the ONLY place many dissertations are ever published, IIUC.

I would not expect the academic world to be even AWARE of most dissertations from unaccredited programs regardless of the quality (or otherwise) of the work itself.

Then, the unaccredited doctorate holder is unlikely to have access to, let alone employment in, the mainstream academic world. He will therefore not have access to faculty appointments and will therefore miss out on funding for reseach and opportunities for publication.

Now, my objections should not be seen as an indictment of our system of accreditation; I STRONGLY suspect that the consequences of pursuing an unaccredited doctorate are exactly what they should be.

However, I DO have misgivings about the validity of the "Google test" as a means of judging the likely quality of any single piece of research.

 

Good questions. Without parameters there won't be results much more informative than what we now know.

Since the comparison is to RA, limiting the initial comparison to US schools would be a feasible undertaking and likely more meaningful than any open-ended, hodgepodge study.