Many studies criticize student evaluations of teaching as biased or a poor measure of teaching effectiveness, or both. But none of these papers are as expansive as a new metastudy of more than 100 articles on these student evaluations, or SETs.

The new study’s breadth means its authors can cut through the sometimes contradictory research on SETs. And instead of looking at just measurement bias (how well SETs reflect good teaching, or don’t) or just equity bias (how SETs advantage certain groups of instructors over others, or don’t), the study contextualizes both.

Co-author Rebecca Kreitzer, assistant professor of public policy at the University of North Carolina at Chapel Hill, said Tuesday that “our conclusions are more nuanced than previous research, particularly on equity bias.” Indeed, where many studies have found evidence of gender bias against women in student evaluations, Kreitzer and co-author Jennie Sweet-Cushman, associate professor of political science at Chatham University, found that the equity bias effect is “conditional,” as “sometimes women and people of color do benefit.”

Yet the effect of gender varies “considerably across disciplines,” with women receiving lower scores in the natural and social sciences compared to the humanities, Kreitzer added. She and Sweet-Cushman also found “an affinity effect,” whereby women tend to prefer female instructors and men prefer male instructors.

Perhaps most important, Kreitzer said, she and Sweet-Cushman found conforming to prescribed gender roles has a more significant effect than gender itself. This is “deeply concerning because students prefer professors with masculine traits, yet penalize women for not conforming to stereotypes.”

All told, Kreitzer said, equity bias exists. But its effect is hard to pin down.

Just as important as its literature review, the new article makes numerous suggestions as to how administrators should use SETs to evaluate professors. Kreitzer and Sweet-Cushman also call out this corner of research for its relative lack of attention to issues of racial and intersectional identity bias, as most of the equity bias research is about gender.