Student Evaluations of Teaching: WHY is this still a thing?

My institution just created a data science major. But it doesn't care about using data in honest and robust ways any more than other institutions.

It's gotten to the point that it's intellectually embarrassing and ethically troubling that we are still using student evaluations of teaching (SET) in their current form for assessing instructor job performance. It is laughable that we do so with numbers computed to two decimal places. It is scandalous that we ignore the documented biases (most especially gender-based). But we do.

Why isn't this an active conversation between faculty and administrators?  I certainly find teaching evaluations helpful - trying to understand why I got a 3.91 on course organization but a 4.32 on inspiring interest is a useful meditation on my teaching practice.  I have to remind myself that the numbers themselves do not mean much.

Telling me where my numbers stand vis a vis my colleagues or the college as a whole FEELS useful and informative, but is it? I THINK I must be doing a better job than a colleague who has scores in the 2.0 - 3.0 range. But doing a better job at what? If you think hard about it, all you can probably take the bank is that I am better at getting more people to say "Excellent" in response to a particular question. The connection between THAT student behavior and the quality of my work is a loose one.

Maybe I am on solid ground when I compare my course organization score to my inspires interest score. MAYBE I am on solid ground when I compare my course organization score in one class to the same score in another the same semester or the same class in another year. I might, for example, think about changes I could make in how I organize a course and then see if that score moves next semester.

But getting seduced by the second decimal place is ludicrous and mad. Even fetishizing the first decimal place is folly. For that matter, even treating this as an average to begin with is bogus.

If you also use these numbers to decide whether to promote me, you've gone off into the twilight zone where the presence of numbers gives the illusion of facticity and objectivity. Might as well utter some incantations while you are at it.

Some new research adds another piece of evidence to the claim that the validity of the numbers in student evaluations of teachers is probably pretty low. Validity means "do they measure what you think they measure?" The answer here is that they do not. Instead, they measure things like "what gender is your instructor?" and "what kind of grade do you expect in this course?"

These researchers even found gender differences in objective practices like "how promptly were assignments graded" and these persisted when the students were misinformed about gender of instructors.

Let's start implementing a policy we can have some respect for. No more averaging. No more use of numerical scores in personnel review. No more batteries of questions that ask more or less the same thing (thus distorting the positivity or negativity of the overall impression).

As John Oliver asks, "why is this still a thing?"

• Colleen Flaherty. Bias Against Female Instructors: New analysis offers more evidence against the reliability of student evaluations of teaching, at least for their use in personnel decisions." Inside Higher Education January 11, 2016
• Boring, Anne, Kellie Ottoboni, and Philip Stark. "Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness." ScienceOpen Research. 2016-01-07
• Stark, Philip B. "An Evaluation of Course Evaluations." ScienceOpen

NPR's Cladio Sanchez' "6 Education Stories To Watch In 2016"

NPR's senior education correspondent offers his predictions for stories in education in 2016.


1. The New Federal Education Law

The long, grueling fight to overhaul the 14-year-old No Child Left Behind law is over, but that'll turn out to be the easy part. The new Every Student Succeeds Act returns most government oversight of schools back to states. But there are no guarantees that the states will do a better job than the federal government in two key areas: closing the achievement gap and raising the performance of the absolute worst schools.

There will be some relief for students burdened by excessive testing. But for the most part states will continue to rely on test scores, using them to punish schools rather than for improving curriculum and instruction. Reading and math scores will drop for all kids on the new, tougher standardized tests linked to the Common Core. But the dismal performance of groups that struggle will trigger more scrutiny from civil rights groups in 2016. We'll also see those groups pressure states to deal with teacher quality and funding.

2. Moving On From Common Core

The controversy over the much-maligned Common Core State Standards will diminish. States will continue their efforts to re-brand or rename the standards, while for the most part following them. Despite the political controversy, the push for high academic standards will continue, and we'll see little of the "race to the bottom" that happened under NCLB.

3. Charter Schools Under A Microscope

The charter school movement will celebrate its 25th anniversary in 2016. With 6,700 schools and nearly 3million students across 43 states and the District of Columbia, charters are a powerful force. The federal government has poured billions of dollars into charters, and polling shows that a majority of Americans support them. But you can expect these publicly funded, privately run schools to face new scrutiny, and new criticism.

We'll see more scandals involving fraud, corruption and mismanagement, despite efforts to weed out "bad actors" who've exploited weak charter laws in several states. As Joe Nathan, a senior fellow at the Center for School Change, who helped write charter school legislation in 32 states, puts it: "We have not done enough to deal with the crooks and charlatans, of which we have our share."

Charters will also be one of the very few education issues to get any attention in the presidential campaign.

4. Dreamers Dreams Deferred

There will be an even stronger backlash against the push for greater access to college for undocumented students. Dreamers — students brought to the U.S. illegally as children — will face greater opposition because of the stalemate over immigration reform. The angry, anti-immigrant rhetoric from Republicans running for president will also shape this debate. Look for state lawmakers to consider even tougher measures to deny dreamers any benefits and push them deeper into a legal and educational limbo.

5. Goodbye Race-Conscious Admissions

Watch for the U.S. Supreme Court to ban race in college admissions, forcing institutions to abandon affirmative action policies. Schools will have to rethink how they recruit and enroll students in efforts to increase diversity. This will fuel an already tense situation on many campuses. Expect minority student protests and campus unrest to intensify.

6. Student Debt Takes Center Stage

Higher education leaders, or what presidential candidate Senator Marco Rubio calls "the higher-ed cartel," effectively killed the Obama administration's attempt to create a more transparent, consumer-friendly way for students and parents to rate colleges. But with many of the presidential candidates calling for tuition-free or debt-free college, we'll see these institutions undertake a more serious discussion about changing their pricing policies — largely out of fear that lawmakers in Washington will step in and do it for them.


Last Year's List

1. Standardized Testing Under Fire
2. More Troubles For The Common Core
3. In Congress, Deeper Divisions
4. Focus On Campus Behavior
5. Teacher Evaluation, Training, And The Vergara Fallout
6. The Ferguson Effect: New Scrutiny For School Police

And some from NPR's "crowd sourced" predictions for 2015.

1. Blended Learning As A Daily Practice
2. More Scrutiny of Student Data
3. Broader Disclosure On Student Loan Defaults
4. Moving On From Common Core Debates
5. Reauthorization of No Child Left Behind; More School Choice
6. Customizable, Game-Like Platforms
7. Transition For The Online Education Space. "Snackable" learning will become a large part of the online education menu.
8. More Options For Student Borrowers
9. Competency-Based Education Picks Up
10. More Nuanced Kinds Of Data In Schools
11. The Digital Classroom Meets Labor Issues

From the "Good Grief!" Department

from Inside Higher Ed

U of California Criticized for Extending Transfer Deadline

December 14, 2015

The University of California announced early this month that transfer applicants to system campuses -- who thought they had to finish applications by the end of November -- could apply as late as Jan. 4. The university said it was acting because UC campuses recently committed to admitting more transfer applicants. For students who still want to apply, this is, of course, good news.

But the Los Angeles Times reported that many of those who met the standard deadline, and the counselors who helped them, are frustrated. Mihai Gherghina, who met the regular deadline, said, "They didn't tell anyone about this extension until after the deadline. It's unfair how some lazy people were given another chance." Adding to the frustration: those who submitted their applications for the early deadline will receive no preference and will not be permitted to edit their applications between now and Jan. 4.

And then there were ...

From Inside Higher Ed

College of New Rochelle Goes Completely Coed

December 9, 2015

Another women's college has decided to go completely coeducational.

The College of New Rochelle on Tuesday announced its plans to begin accepting men into its School of Arts & Sciences in fall 2016. The New York college has been accepting men in other programs for about four decades. Its School of Nursing, School of New Resources (for adult learners) and Graduate School are already coed -- the college's School of Arts & Sciences was the last holdout, and has been women only since the college was founded in 1904.

“This decision was made after very careful thought, evaluation of several key factors, and above all with a great reverence for the college’s mission,” Elizabeth LeVaca, chair of the college's governing board, said in a statement, adding that the board received supportive feedback on the change.

A Facebook page for New Rochelle alumni contained a mix of comments, many supportive and understanding but several quite critical.

Will Free Community College Put HBCUs Out of Business?

My colleague Sara Goldrick-Rab wrote a post saying "Short answer: No." But notes it might be because "HBCUs (both public and private) are allocated $10Billion in support under America's College Promise."  I confess to ignorance about the details of ACP, but it seems like something we should be paying attention to at tuition-driven SLACs.

See also AACC on America's College Promise Act and this brief from democrats on the Committee on Education and the Workforce. Are the potential grants to Hispanic Serving Institutions driving some colleges to try to redefine their mission?  Here are a few notes from the above document:

In order to be eligible, MSIs must have a student body that is at least 35 percent low-income, including Pell-eligible students. Additionally eligible MSIs must commit to maintain or adopt evidence-based institutional reforms designed to improve student outcomes, and to set performance goals for improving those outcomes. Eligible MSIs that enter into articulation agreements with community colleges can also receive grant funds for eligible students who transfer from those community colleges to complete their baccalaureate degrees.

This is intriguing, but one would expect a bit of careful analysis about the costs, benefits, and implications of chasing this not-yet-existing funding.

Wisconsin Center for Education Research

Can Free Tuition to Community Colleges Put Historically Black Colleges and Universities Out of Business?

T. Ramon Stuart, Ph.D.

Associate Provost and Associate Vice President for Academic Affairs,West Virginia State University

December 4, 2015 10 am - 11:00 am

During the 2015 State of the Union Address, President Barack H. Obama announced his vision to provide American citizens free access to higher education through community colleges. While President Obama failed to outline the details of his plan, there is no doubt that his plan could drastically increase the number of Americans with a college degree; however one very important detail that President Obama omitted from his statement was the cost of this initiative and the impact that funding it would have on other institutions of higher education - especially Historically Black Colleges and University. This study uses current IPEDS data to analyze the tuition cost of the 1890 land-grant institutions while also exploring the average tuition cost of the community colleges in the respective states to see if there is a substantial difference in tuition cost.

MOOC on Universal Design in February

Accessibility: Designing and Teaching Courses for All Learners is a free 6-week professional development course available that will help you gain a better understanding of accessibility as a civil rights issue and develop the knowledge and skills you need to design learning experiences that promote inclusive learning environments.

During this six-week course, you’ll learn how to:

●     Recognize and address challenges faced by students with disabilities related to access, success, and completion.

●     Articulate faculty and staff roles in reducing barriers for students with disabilities.

●     Apply the principles of Universal Design for Learning (UDL) in designing accessible learning experiences.

●     Analyze the benefits of Backward Design when developing learning experiences.

●     Use Section 508 standards and WCAG 2.0 guidelines to create accessible courses.

●     Determine which tools and techniques are appropriate based on course content.

You will have the opportunity to earn badges that recognize your mastery of these competencies.

Audience: Anyone may enroll and participate in the MOOC. It has been designed for faculty and staff in higher education at any type or level of institution.

Why take the Access MOOC? Watch this short video to find out! (Audio described version)

Access MOOC begins on February 22^nd

Next steps:

1.       Register at Canvas Network

2.       Share and follow the conversation on Twitter using #AccessMOOC

3.       Follow our Access MOOC Facebook Page

The course is a collaborative effort of faculty and staff from SUNY Empire State College and SUNY Buffalo State College, funded by a SUNY Innovative Instruction Technology Grant.

How Does This Figure in Our Strategic Plans?

Our institution is basically a California school (~82% UG, ~83% G). Four other states send us a over 9 undergrads but all the rest are single digits. By contrast Berkeley and UCLA have over 20% out of state.

On the one hand, that suggests that we are more of a California state school (or less of a national school) than are some state institutions. On the other hand, it's good for our bottom line if California students' places at UCs are taken by out of state students.

But now that will change.

According to the UC Newsroom: "The University of California will significantly boost enrollment of California freshman and transfer students next year.

The UC Board of Regents today (Nov. 19) approved a budget plan to enroll an additional 10,000 California undergraduates over the next three years, including 5,000 freshman and transfer students in 2016-17. 

Under the plan, all nine UC campuses that educate undergraduates will enroll more California students.

Insofar as our business model includes picking up qualified students who do not get into Berkeley but want to study in the Bay Area, this good news for California's students might be disconcerting news for us.  Five thousand students over 10 campuses might mean only 500 at UCB and only half of them women. But say 1 in 20 of them would have considered Mills: that's maybe a quarter million in net revenue.

There might be a glimmer of a bright side: the plan also includes adding up to 600 more graduate students to the system - if we can be a producer of same, options for our alumnae increase. It's a safe guess this won't compete with our graduate programs since most of ours are probably not in the high undergraduate demand column.

The absence of things like this (that is, external policy changes that can have outsized impact on what we are doing - the "free" community college initiative, for example) on our radar screen suggest (confirm) a dangerously parochial perspective and do not bode well for future success.