Incompetence and Assessment

by Cliopatria

My father sent along a link to a article: he didn't think much of it, and was actually sending it along as an example of a self-evident proposition the empirical demonstration of which required extensive effort and for which someone got academic credit. Goodness knows there's lots of that out there: eating too much junk is bad for you, as is sitting around doing nothing but watching TV; that sort of thing (as my father says,"anything, in sufficient quantities, will kill you"). My personal favorite, also sent along by my father, I think, was the research showing that children who scored high on" curiousity" as three year-olds performed better in the first two years of school than children who had scored poorly; the Boston-based researchers did the research on children somewhere in the Caribbean, obviously requiring regular return visits, probably during the colder periods of the school year (in-class observation had to be part of the process).

This article, though, is much more interesting: Justin Kruger and David Dunning (Department of Psychology, Cornell University),"Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments," Journal of Personality and Social Psychology 77:6 (Dec 1999), 1121-1134. The abstract concludes [emphasis added and some statistical notations removed throughout]

Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.

"In essence," they continue,"we argue that the skills that engender competence in a particular domain are often the very same skills necessary to evaluate competence in that domain–one's own or anyone else's." This isn't actually terribly original: they cite a pretty good stack of previous studies demonstrating that deep incompetence affects not just performance but evaluation. Even grading cannot improve the situation short term:"bottom-quartile participants failed to gain insight into their own performance after seeing the more competent choices of their peers. If anything, bottom-quartile participants tended to raise their already inflated self-estimates, although not to a significant degree." People can learn to recognize their level of incompetence:

In fact, the training packet was so successful that those who had originally scored in the bottom quartile were just as accurate in monitoring their test performance as were those who had initially scored in the top quartile. ... To be sure, participants still overestimated their logical reasoning ability, and their performance on the test relative to their peers, but they were considerably more calibrated overall and were no longer miscalibrated with respect to their raw test score.

Still, without tailored pre-training (which didn't actually raise the test scores much), incompetents remained very poor judges of performance.

In their analysis section the authors ask"Although our analysis suggests that incompetent individuals are unable to spot their poor performances themselves, one would have thought negative feedback would have been inevitable at some point in their academic career. So why had they not learned?" Some of the answers are predictable: politeness, mistaken attribution of failure to other factors, lack of self-correction opportunities, selective self-awareness and relative lack of transparency of the success of others. I would add two factors which they do not address, related to the structure of academic enterprises: social promotion (including higher-ed "retention") and the use of student evaluations in hiring/retention/tenure/promotion decisions.

To some extent, this study is a demonstration of something that I've been saying for years: experimental psychology is the study of undergraduates, particularly psychology majors: in this case, Cornell students getting extra credit in psych. On the other hand, that more strongly supports my reaction: Quantitative student feedback is deeply flawed data and should be used only very carefully. Minor differences in scores are meaningless; major differences in scores may be meaningful only if outliers and course level and other known independent variables are controlled for.

Something to think about while we're handing out those [censored] bubble forms....

comments powered by Disqus

More Comments:

chris l pettit - 11/29/2004

You know, one of the first things you learn in Jury Selection (yes there is a whole course on it in law school...sigh) is to figure out how to pick what usually ends up to be the least educated, most ignorant (Buddhist definition), lower middle class to upper lower class, white (unless they are very anti-Castro...or a disgrace to their race like Clarence Thomas) jury...so that the trial becomes an oratorical contest. law does not figure...the acting abilities of the lawyers do. It is actually quite a funny class...and the actual theories behind jury selection would make a hilarious short movie of sorts (if anyone is up for money making ideas). I thought it was hilarious when I was doing homicide investigation, throughout law school, and continue to through present times. THere are many instances where the "will of the people" needs to take second nature to tightly regulated indivduals...namely in the areas of human rights and the rule of law. To look at the South African Constitution...the judges are appointed by a judicial commission representative of nearly all aspects of society that is (at least at the moment) pretty free from political bias. In addition, written into the Constitution is an obligation that the judges uphold human rights and respect (take into consideration in most cases) international law and opinion. If the justices fail in these regards, one can make a case to the African Court of Human Rights and, if the case goes against the judges...those judges must be removed. Seeing as though the most prestigious and impartial scholars are put on the ACHR, this seems like a pretty good system. Of course we in the US would not hear of it...to "non-democratic"...as if we know what a democracy is anyway.

My question is what to do when you encounter someone (as often happens on this website) who is of the ilk of the samples described in the article. THey most often cannot be countered by logic or reason...as they are usually arguing from a fundamental philosophical flaw that entails some sort of what could be called blind faith. Debating them is like bouncing a rubber ball off a wall. I heard a saying one that arguing with an idiot is futile because they force you to come down to their level and then beat you with experience. how true it is in many instances. is our only option to address it in the primary and secondary school phases? Education seems to be key...

By the way, the Mind and Life Institute that I am a member of has a series of great articles on the neuroplasticity of the brain that are interesting to read in this context. The Institute is basically a meeting of the minds between leading practitioners of the science of the mind component of Buddhism and leading Western scientists in neuro-biology, quantum theory and physics, and other sciences waaaaayyyyy beyond most of us normal mortals. What is fascinating is that the studies conclusively show that a) Buddhists are happier and more neurologically developed (sorry...little promotion and satisfaction...you can ignore that) b) that one can easily change the structure and functions of the mind through mental exercise, self examination on a daily basis, and meditation. THis includes examining ones viewpoints and the fallacies that they are built upon. The only problematic feature is that even though this is possible, it is still dependent upon the practitioners to actually engage themselves instead of wallowing in their ignorance. How do we get people to engage themselves? Is it again rooted in primary and secondary education (my mentors seems to think that it is)? How does one introduce it in the adult sectors since they are the ones usually dictating the affairs of the world and the workings of the education system.

Anyway...a link to the latest article for you guys...i find this stuff fascinating...

http://www.pnas.org/cgi/content/full/101/46/16369

CP
www.wicper.org

mark safranski - 11/29/2004

" I tend to agree with your point that teaching content is crucial to skills development, but not to the exclusion of an awareness of transferable skills teaching, pedagogic and developmental theory, etc."

Oh, I wouldn't exclude methodological courses, but at the BA/BS level they should be adjunctive, not a major. At the MA or doctoral level, where the student is usually getting a degree in order to be in an administrative role supervising and mentoring other teachers, methodology is a worthwhile focus.

I just believe that prospective teachers should themselves have to grapple with the difficult content-related thinking required by upper-division courses before we unleash them on students. The particular field matters less than they have that experience and a base of knowlege to go along with the skills.

Jonathan Dresner - 11/29/2004

Actually, I think there is grounds for some hope in the article. (Though the article only really proved that teaching logical thinking improved self-assessment on logic tests) The structures which preserve incompetence can be removed. If "politeness, mistaken attribution of failure to other factors, lack of self-correction opportunities, selective self-awareness and relative lack of transparency of the success of others" are the problem, then we need to provide realistic feedback, assignments and environments which allow students to focus on specific skills and challenges without complications, opportunities to apply feedback and learned skills to demonstrate improvement, programs which track students better so that each class is not taken in total isolation, and a more open environment in which success and failure can be addressed.

It's not impossible. It's just difficult and requires structural change.

I tend to agree with your point that teaching content is crucial to skills development, but not to the exclusion of an awareness of transferable skills teaching, pedagogic and developmental theory, etc.

mark safranski - 11/29/2004

The alarming part of this study, which I read when it came out, iis the degree to which incompetence appears to be intractable. Incompetents are, if not impervious, quite resistant to reality in the form of seeing cause/effect relationships. Which should give anyone pause when you consider the jury system.

If I recall properly, the study indicated that explicitly teaching logical thinking seemed to help moderate the degree of a subject's incompetence. We should strongly reconsider the value of populating elementary and middle schools primarily with teachers with degrees that required few or no upper level classes in science or math. Or for that matter, History.

Since the vast proportion of this cohort of educators are essentially educated in a *process* - as reading specialist/education majors - and American school children remain remarkably poor readers, we might ponder the value of *content* in creating effective educational systems.