Another Thought on "Rigor"

This is a long post - but writing it has allowed me to summarize some thoughts I've had. I'm blowing off some steam here, but here goes...

We've been debating the topic of "rigor" in DL PhD programs. It strikes me that there is a reason to talk about this topic. How many posts have there been asking, in essence "can I earn a DL PhD and teach in a university?".

Talking about "rigor" leads to a number of challenges:

1. What is "rigor"? Can we define it? Or is it something we "know when we see it"? Is "rigor" the same as "strictness", "severity" or "harshness" as my thesaurus says?

2. Generalizations about individual or groups of schools are difficult. There is likely to be variation between schools (e.g. NSU and Harvard or NSU and University of "small state"), not to mention within different departments in a given school (e.g. School of Education versus School of Business) and between different students. The variation at one school between the best and worst dissertation may well be greater than the difference in mean "rigor" between schools.

It would be great to collect data and perform some sort of analysis. However, I doubt any of us have time to do this. Even if we did, what objective measures would we use? The one thought I have is to look at the results of DL dissertation research, but even this idea has some problems.

My own biased basis for comment on this topic includes completing a DL dissertation (from NSU), participating in several dissertation committees (at NSU and two other DL schools), working with DL graduates from four different schools and reading a fair amount of dissertation output - e.g. conference papers and journal articles. I certainly don't know everything there is to know about this topic. In particular, I've not worked with any students from UK or Australian school (hence, I'm only talking about US DL schools). However, I'll go out on a limb and suggest the following:

Take 100 randomly selected dissertations. Evaluate them for "rigor" (whatever that is). Rank them from 1 to 100. What would you find?

1. I suspect that the top dissertations would probably come
from top B&M schools (whatever that means). Some good
dissertations could come from mediocre B&M schools. I
doubt you'd see any DL dissertations in the top 30.

2. I suspect that the DL dissertations would show towards the
bottom of the ranking - mixed in with some work from lesser
B&M schools.

Perhaps someone could test these ideas - I know I don't have time to.

My interest in this topic is that I'm quite concern about standards in higher education. I'm aware of a number of schools that are adding part-time PhD/DBA programs. It seems that some schools have decided that such programs are a "growth segment" for them.

This would be ok if standards are maintained. But I'm quite concerned that in the rush to add DL PhD/DBA programs (and students and tuition) standards may fall by the sideline.

Let me relay a story that shows the concern. Recently, I was talking with a peer at another school (not discussed in this NG) that recently added a part-time DBA program. At first the school started their doctoral program without a GMAT requirement. They found they had real uneven quality in their students - some did fine, others were pretty awful.

So my friend pushed to require the GMAT among incoming students. This year they reviewed their applicants and noticed that they had some candidates with GMAT scores in the bottom 5 percentile. He noted that these candidates had MBAs from part-time/DL institutions with high GPAs.

Let's get real frank here. I know the GMAT isn't a perfect test and that some students test better than others. Further, minor variations in scores are not significant. There is effectively no difference between, say, a 520 and 530. But bottom 5 percentile on the GMAT? That's 2 standard deviations below the mean. How could a person earn an MBA in a credible program - and score this low? Something is wrong here.

What I see coming out of DL PhD/DBA programs is a wide variation of students - some do fine. For many students, a DL PhD/DBA program provides access to graduate study that they are qualified to pursue - but can't do so at a traditional B&M school.

But I'm very concerned about a new breed of marginally qualified students that enter such programs. Encourage by non-rigorous DL MBA programs they ask "why can't I earn a PhD"? I'm not convinced that DL programs - especially the for-profit ones - are willing to say "no" in admitting students or "please leave" to those that can't do credible work. Worse yet I see a day when DL MBA programs become populated with even more DL PhD/DBA graduates as faculty members then there are now.

My answer will never fly in "for-profit" and "new wave non-profits". First, DL programs need to require the GMAT or GRE (as do virtually all credible B&M PhD/DBA schools in the US) and they need to have the guts to say "no" to students that clearly aren't qualified. Without a 500 GMAT/GRE you won't likely gain admission to any PhD program at a B&M school - why should you get into a DL program? Second, DL programs need to maintain high standards - and be willing to say "no diploma" to a "customer", even though the customer paid his/her money.

Regards - Andy

 

The only reason we are discussing this issue here is because distance learning is relatively new in the United States. These discussions probably (I say probably) may not have any merits in the United Kingdom and even in English speaking African countries where distance learning have been as old as education itself.

While growing up, I saw so many people earn high school diplomas called GCE for General Certificate of Education (GCE) went on to colleges and performed excellently well. I saw individuals earned college degrees from Universities in the United Kingdom while never setting foot in United Kingdom. These individuals went to be poets, statesmen, judges, Medical Doctors, PhDs, Business Executives and so on.

The concept of distance learning has not changed. What has changed is the mode of delivery. Instead of waiting for the mailman, you can now get your education at the click of a mouse. The United States being as innovative as it is has taken this new delivery method and is essentially perfecting it, perhaps more so than any other country on earth. That is not surprising.

To generalize that only B & M schools have some rigor in its education carries no weight. It is like saying only AACSB graduates get good jobs or may teach in ACCSB schools. These are equally nonsense. Based on my personal experience, those of my family and friends, the only education you require is a regionally accredited degree - mode of delivery is inconsequential and you will get a job - yes with very high figures too. If it works for immigrants that came to this country with no money and have made it, it can work for any body. I have said this in many of my past contributions to this forum.

All these classifications that carry no merits have the capability to discourage those who may not know better from pursuing a decent education and I am on a lone crusade not to let that happen as much as possible.

All these rankings, top tiers, low tiers and so forth are meaningless. You cannot possibly rank schools and programs. Again my friends and family members graduated from state schools and some private accredited colleges and schools. They are all employed in big five accounting/auditing firms, medical practice, hospitals, governments and so one. Never have they been asked about rankings. For accounting, the question is do you have CPA, for a nurse, do you have RN license, for social worker, do you have your license and so on.


All the rankings, residential, non-residential, DL, B&M, AACSB, ICBE, and any other alphabet soups are inconsequential. None of my family and close friends with a college degree is unemployed. A young cousin who just graduated this May already has an offer.

Let us concentrate on driving out unaccredited schools from dishing out diplomas with little or no work. May be I am doing something wrong, I am working very had at my Touro Health Science degree program just like I did for my B&M schools. When I go to class live online, it is just like being in a B&M class except that my Professor is in California and I am in Maryland.

You know what? Today's businesses be it government, businesses and in health care are increasingly being conducted online. We now have telemedicine, teleconferencing and even telecommuting. What is happening is those who are afraid of change will want to denigrate a good thing. Employer intent of not hiring you will find something to say for its decision.

If I were a certified accountant and wanted to teach in a school, if that school won't hire me for not being AACSB graduate, fine, I will move on to another school. At the end of the day, it is contact, (I concede you get more of that at Ivy leagues), hard work, being the best or one of the best in your chosen field that will get you a job.

Many of the things I read on this board especially with respect to getting a job are simply laughable as I have not experienced them. You get job because the market wants your profession not necessarily where you went to get the education and CERTAINLY not the mode of delivery such education. I cannot emphasize these enough.

oko

 

Hi Andy,

Your post addresses your concerns and your suspicions, and you cite anecdote that might have relevance.


What is the correlation between any score in the GMAT, and success in a graduate program? With what degree of confidence can we make a decision to accept or reject a candidate based upon these scores? With all else equal, what marginal benefit do these scores provide decision-makers?

You have obviously thought long and deep about these issues. Other than anecdote supporting predisposition, where is the evidence that your fears are justified?

If we accept for the moment that your concerns have merit, given the lack of any corroborating evidence in your analysis, how do you know that the introduction of the GMAT filter would have any value? Is this to be a faith-based initiative?

You say elsewhere on a related note, that few would have time to collect data and analyze results, and that is surely true enough. Yet, on the specific issue of the relevance of the GMAT and its effectiveness as a predictor of future performance in graduate programs, there are result from many studies only a mouse click away upon which you might have drawn, are there not?

I respond here because you have voiced concern about quality, and caliber, and especially - - rigor. Is there not a certain irony in opining on those concerns in a treatise where that virtue is demonstrably absent?


I do not intend to be harsh for the sake of it, but it has often been the case that a contributor will inaugurate a thread and lament a deficit others suffer that they themselves seem to share in spades.

.

 

Andy Borchers wrote:

Sorry, but I haven't read your post on this thread, but before I do so I just want some further clarification on your quote.

In other threads you described part-time at the doctoral level as basically working full time and taking courses on the side regardless of the credits taken, right? And what you meant about "full-time" is basically total dedication to a doctoral program without any outside distractions or commitments (work). If thats the case then you are saying that ALL distance doctorates are part-time programs and because of that none of them are perfect substitutes to full time programs whether they are from strictly online schools or from traditional B&M schools (such as the University of Nebraska-Lincoln).
And what about traditional doctoral programs taken on a part-time basis ? are we going to say that they are no substitute for attending the same program full time?

I'm not saying that I agree or disagree with your quote, all I want is some clarification on what you meant with it.

-S

 

Re: Re: Another Thought on "Rigor"

Lawrie - I have the following to offer:

1. In personal conversation with a former director of the NSU DBA program, I was told that he found a significant correlation between GMAT scores and completion rates for dissertations.

2. The NSU DBA program went through a period where they admitted students without a GMAT score. Instead, they permitted students to submit a portfolio. The school changed their policy and returned to requiring the GMAT. The portfolio approach didn't meet their need for appraising candidates as well as the GMAT did.

3. At another school (nameless and generally unknown in this forum), a peer has reported to me his experience in watching three cohort classes of doctoral students come into the program. The school started without a GMAT requirement - and have now switched to it. They found utility in using the GMAT at a "gross level" to identify weak candidates.

4. The GMAC has studied the issue of GMAT score and first year MBA grades repeatedly. There is a statistically significant correlation between GMAT and first year MBA grades. Perfect correlation - of course not. Statistically significant? Yes. One could say - "What do first year MBA grades have to do with doctoral study?" The two aren't the same - but they would certainly seem to be related.

According to the GMAC website:

"The predictive validity of the GMAT has been established by analyzing the statistical relationship between GMAT® scores and first-year (or midprogram) grade point average. ... The average (represented by the median) statistical relationship is +0.41 out of a possible 1.00. When undergraduate grade point average was added to the three GMAT variables the predictive power increased to +0.47"

5. The GRE has been studied extensively as a predictor of success in graduate education. The results would appear to support the use of the GRE as a incoming "screen" for graduate education. As with the GMAT correlations are significant, but far from perfect. See the ETS website:

http://www.gre.org/respredict.html#TheRelationshipofGREGeneralTestScores

So - do I have complete data to back up my statements - no. I'm sure one could devote their life to studying this topic. But is this some "faith based initiative"? Hardly - there is data to support what I'm saying.

Ultimately, readers of this NG will have to draw their own conclusions.

Regards - Andy

 

Oko,

I couldn't agree with you more, any accredited school will do it for you if you want to get a typical industry job. However, an academic job is an entire different animal. Academia is perhaps one of the few places where people would actually care where you went to school. I'm only a contract non-tenure lecturer and the first thing collegues ask me is, where did you go to school?. Also, academics in general are afraid of virtual education and not very keen about it. They also feel that if they went for four or more to get a PhD you should pay your duties too.

 

Sulla - You raise a good question here. There are a wide range of doctoral programs and arrangments for study. The best regarded doctoral programs in business (e.g. Michigan, MIT, Stanford) almost always require full-time dedication to studies and reserach for four or more years. I know that's what I found out when I looked for programs in my state (at UofM and MSU).

At the other extreme, the US DL programs we talk about in this forum are designed for and taken by students who, in most cases, are employed. Often the students that pursue such programs have families and other obligations as well. In my time at NSU's DBA program I don't remember a single student that was totally dedicated to doctoral studies.

I think it is great that DL programs make doctoral studies possible for mid-career folks like me. My only caution, however, is in comparing traditional full-time B&M programs with DL offerings. If full-time programs require 4-5 years - how can a student working full time in a career and caring for a family complete a comparable program part-time in 3-5 years? It sounds to good to be true.

In between the extremes are all sorts of variations. Some students at B&M schools may be part-time. Others may be students who also serve as teaching assistants. Others may be part-time for part of their study, but may be required to meet full-time status for part of their study. For example, at the University of Toledo students in their PhD in Manufacturing Management are required to spend 3 quarters in residence, while the rest of their study can be part-time.

As for my statement - I'm going to avoid the variations in the middle. I'll maintain that on average the rigor of full-time B&M doctoral programs is greater than that of DL doctoral programs. Hence, the two are not perfect substitutes.

Regards - Andy

 

OKO – I'd like to add an observation to your response. First, I applaud your faith in the US and I hope we continue to serve as a magnet to the gifted of the world. I'm first generation and one of the messages I heard loud and clear while growing up was the opportunity my father experienced in this country. I am reminded daily in my workplace where I practice alongside and train a number of international medical graduate resident physicians of how fortunate the US is to attract the best and brightest. In my profession, which is psychology, the “alphabet soups” are not inconsequential. Graduates of non-APA-approved programs are marginalized as, I suspect, are graduates of non-NLN nursing and non-CSWE social work programs as are non-ABA trained attorneys.

I'd also like to comment on the issue of the academic market. In my experience landing a job at a top university isn't just a function of what school the applicant attended. It has huge amount lot to do with the applicant's publication record. The student who has an eye on a tenure track position at a top university is well-served to connect with an adviser who has an active research lab so that he or she not only produces a top notch dissertation but numerous publications in refereed journals and presentations and poster sessions at top conferences along the way. Not to mention having a powerful, well-connected mentor. Mine is a B&M degree so I don't know how things work in the world of the online doctorate but my suspicion is the structure may not readily lend itself to the intangibles like the Friday after work beers and the bonding that takes place f2f and on road trips to presentations and conferences in addition to a structure that lends itself to research that culminates in access to the academic brass ring. Andy, I wonder, is this part of what causes you to identify online doctoral training as “niche?”

 

I'm not certain about the definition of "rigor" here. But I believe that many of Andy's assertions would be correct if, instead of "rigor," one considered the extent to which the doctoral research contributes to the body of knowledge in the field. One possible way to assess this would be to use citation analysis (as established by Eugene Garfield). This provides a way to quantify the impact of the work by tracking the number of times the publications that arise directly from the doctoral research are cited in other journals (and each journal itself can be weighted by its own citation impact). There are pitfalls in this approach, but the methodology (which has been refined over a period of more than 3 decades) has proven useful and is quite widely employed.

 

Talking about "rigor" leads to a number of challenges:

1. What is "rigor"? Can we define it? Or is it something we "know when we see it"? Is "rigor" the same as "strictness", "severity" or "harshness" as my thesaurus says?




But I'm very concerned about a new breed of marginally qualified students that enter such programs. Encourage by non-rigorous DL MBA programs they ask "why can't I earn a PhD"? I'm not convinced that DL programs - especially the for-profit ones - are willing to say "no" in admitting students or "please leave" to those that can't do credible work....

...DL programs need to maintain high standards - and be willing to say "no diploma" to a "customer", even though the customer paid his/her money.

Regards - Andy [/B][/QUOTE]

===


I appreciate your opinion. I have similar concerns for Biblical Studies at the doc level.


IMO in that discipline rigor or the lack of it at the dissertational level, however, is not difficult at all to identify. Rigor here has as its correlates scholarly interaction with a particular body of knowledge and the production of a worthwhile addition to that knowledge. This is not at all hard to measure!

Also neither IMO while a DL program in Bible may allow, for financial reasons, unqualified students into a doctoral program ,and may produce graduates who grossly fail to measure up to commonly held standards in doctoral studies in Bible , it just need not be that way.

Prerequisites to entering such a program should include an accredited/GAAP masters in which a scholarly thesis was done. This thesis should evidence an advanced knowledge of the general area of study , the ability to work in whatever languages are appropriate to the study , and the aptitude to do scholarly research .

I'm sorry that I do not have examples of DL vs B&M dissertations in Bible to provide evidence for my opinion that in Biblical Studies it need not be that one format is more likely or less likely to produce a better or poorer dissertational experience in terms of rigor.

But nevertheless IMO it is not difficult to measure one's own DL progress in terms of rigor by an honest comparison of one's own research with the scholarly research as found in the many theological journals and in the standard tools of the field . Some of these journals and tools are very technical, so if one's own research is genuinely comparable with that, then, IMO, such is an indicator of rigor being done. It may not hurt either to have one's chapters, as I do, also reviewed by a B&M PhD who is regularly published in the field.

Keep up the good work, Andy.

 

Andy,

I totally agree with your assessment of the situation at the for profit DL schools.

I received my BBA from a DL school that did not require the GMAT and, for my initial purposes, that seemed like a great idea. I just wanted to get a job.

But I decided I wanted to get an master's in business and chose Nova to do it because it did not require the GMAT. I decided to go ground instead of DL so I could make contacts and meet people - big mistake. I found that the caliber of many students was disappointing - Although there were many exceptions. The lower students would try to do minimal work (especially in groups) and did not much care about getting excellent grades.

The following semester, I switched to the eMIBA program.

I am now considering the Doctoral program at Nova. To that end, along with my regular course work, I am brushing up on my math skills (Algebra - Calculus) in preparation to take the GMAT. It may not be required, but I want to give myself the best preparation possible for future research and give myself better standing.

Bottom line is, many of the students I met at Nova may not have gotten in if a GMAT was required. At the very least, requiring a GMAT would make a student prepare for a test that may start their cerebral juices going prior to starting class.

Nova's a great school with awesome professors but like all schools, it's the student's personal achievement that makes the difference. As for rigor, I do not know first hand about Nova's DBA program, but the master's level courses are very thorough and, I believe, challenging.

 

Ah, now this is a horse of a different color. The term rigor is simply too vague in its universally accepted metrics.

I do accept the argument quite easily that BM programs better prepare people for jobs as tenure track faculty in other BM schools. I also accept the argument that, relative to scholarly, peer reviewed published works, BM programs produce a higher percentage of such than DL programs.

Now, having said that, I see no necessary correlation to "rigor" and the above two areas that I would agree BM leads DL.

 

maybe not, but the online psychology doctorates that are APA (Fielding, CIIS) or pursuing APA (Capella, Saybrook), require one year of residency. During the residency periods you have the opportunity to network and bond with other students, and attend local conferences.

Most of the online psychology doctorates follow the practioner-scholar model (PsyD) than the scientist-practitioner one, so publications are not going to be the main focus. And many if not most of the student's goals are pursuing clinical practice as a licensed psychologist rather than a career as a tenured research professor. The only APA approved school mentioned above that seems to follow the scientist-practitioner model is the PhD program from the Fielding Institute.

But you're right in that for anyone seriously considering a tenured-track position at a research I university, publications are the key.

-S

 

Very interesting approach, is there any online tool that allows you to track this?

 

When I read all this talk about "rigor", I can't help but think of corpses. I guess rigor best equates to subjective difficulty. There's also the issue of objective academic standards, which isn't necessarily the same thing.

I think that a lot would depend on one's scholarly interests as well. How are good dissertations distinguished from bad dissertations? How does this good-bad variable correlate to a dissertation's interest? To its value to another scholar who cites its conclusions?

I can imagine general criteria for dissertation quality, but those criteria might not be very relevant if one is looking for a dissertation that treats a particular aspect of cell membrane physiology or something.

Sure, another university might be turning out "better" dissertations, but if they are of no interest, who cares?

Oxpecker made the excellent suggestion of quantifying citations. There are already sources out there for that kind of information I believe, at least in the sciences.

But I'm not sure how useful that data would be. It would obviously favor the stylish academic subjects and penalize those researching obscure or unusual specialties. I may be mistaken, but citations have a slight odor of herd-instinct to me. Some academic specialties are small, pursued by only a few scholars around the world. These people read each others work, but the result isn't a very big blip on the citation radar screen. But that doesn't imply that it isn't valuable stuff. I imagine that some problems in advanced mathematics might satisfy that description.

Ihave some problems with that.

First, there's the universe of discourse. Are we selecting from all dissertations in all disciplines? Or from dissertations that treat a particular research topic? The answer to that question will have a profound impact on our measures of "rigor", I think. The broad universe demands generic quality criteria that might be too blunt for some purposes.

As to predicted results, I agree with Andy I think. But with some minor caveats.

One of the leading programs in the US, whether B&M or DL, with a Jungian approach to mythological studies is the Pacifica Graduate Institute near Santa Barbara. Hell, if your interest is 'deviant' sex practices, San Francisco's CA-approved Institute for the Study of Human Sexuality is probably putting out some important work that can't be ignored. There are a handful of niches.

But yeah, I generally agree that in virtually every field, B&M programs put out 99 + percent of the research that relevant scholars consider important. DL isn't even on the radar screen in most cases.

Of course, DL doctoral programs don't even exist in most disciplines, and are either rare or extremely new in the others. Where they do exist, they seem heavily weighted towards practitioner-oriented continuing-education in fields like psychology and education. So lack of scholarly impact shouldn't really be surprising.

I'll repeat what I've written before: I don't think that DL doctoral education will have arrived until there is work coming out of DL programs that scholars in a field or specialty can't afford to ignore. DL programs need to turn out work that scholars have to read in order to stay current. For me at least, that's the bottom line.

 

First, I'd like to say that this is one of the better threads I've read recently. There have been some very thoughtful statements and questions by a number of posters. Thanks to Andy for starting it off.

There have been comments made regarding the process (coursework, research, etc.) and there have been comments made regarding the "product" (the dissertation and subsequent citations of the dissertation). I'd like to say that another aspect that might shed some light on this issue is the "admissions" issue, that is, who gets in to these programs. This was touched upon by Andy in his original post. I'm referring specifically to his statements regarding the GMAT entrance requirement. I'd be interestd in knowing the acceptance rate (or refusal rate) for B&M residence PhD programs versus DL PhD programs. My experience is that in the B&M programs, many people get turned down. Admission is very competetive, especially in schools/departments that have reputations for excellence. Conversely, (and this is speculation on my part) I expect that most DL PhD programs will admit virtually everyone that meets the basic admission criteria. I'd guess that there's almost no competition for slots in these DL programs. If I'm right, then I'd say that DL PhD's will not be considered to be the equal of B&M PhD's until people start getting turned away by the DL programs.
Jack

 

That's true.

 

I think you're right about the open admissions policies of DL doctoral programs. They exist to meet a need not being met by traditional schools. Theirs is a mission of inclusion, not exclusion. If potential students want rejection, it is plenty available already.

That said, I don't know if admissions is the issue. Students can be excluded for all kinds of reasons, many not related to academics or potential for success. Space, intended research area, cost, access to the program, and many others. We won't know the answer to this question until outcomes, not inputs, are measured.

I suspect traditional universities have high standards. But I've seen some really bad dissertations emerge from that process. DL programs tend to cater to working professionals, most of whom are primed to make serious and significant contributions to their professions (not "fields of study, professions. Most are already active in their fields, not hoping to become so upon finding a job.)

I would make the case that DL schools don't just solve the access problem associated with pursuing a doctorate while balancing family and work, they open doors for outstanding contributions to society, contributions that might not be made otherwise.

Again, admissions are irrelevant to the case I just made. But outcomes are essential.

 

Jack Tracey wrote:

Yes, but B&M doctorate institutions far far outnumber DL doctorate schools to begin with. We are basically comparing over 248 B&M doctoral universities according to USNEWS to over 7-9 DL schools (Walden, Touro, Capella, Union, NCU, NSU, UoP, CIIS, Saybrook, Fielding). Four of which are proprietary and the rest non-profit.
High entry standards are commonly seen in BM schools but are not always the norm. There are BM programs and schools that are incredibly hard to get into (clinical psychology, business, med school) and there are those only require the minimal requirements for entry (especially in programs that are in lower demand-- library science, arts, chemistry, English Literature, etc). And the later far outnumbers the number of DL schools in the US.

Again, none of the DL doctorate schools are particularly picky, but like Rich said, they target accomplished working professionals (I think the average age at Capella is about 45 yrs.) who more often than not are already highly educated (many come in with existing masters from state schools) and are serious about school and making significant contributions to their profession.

-S