Why I Have Never Visited My Rate My Professor Page
Introduction
When I attended Carnegie Mellon, the Internet existed but World Wide Web was just getting started. Students could complain about their professors but hardly in a way that would catch on. Back then, what where were aggrieved students going to go? What were they really going to do?
Today, they go to Rate My Professors (RMP)—and have for nearly 20 years now. I’ve never been to my own page and, in this post, I’ll explain why I’ll never look at it.
Sources of Student Feedback
Generally speaking today, students can provide professors feedback in a number of ways—some direct and indirect:
Direct: Through in-person communications and university-sanctioned semester evaluations
Indirect: Sending anonymous messages, reporting a professor’s behavior to a university office, department chair, by submitting feedback via third-party sites
RMP clearly falls into the latter bucket. Of course, that alone doesn’t reduce or invalidate its value. RMP’s following limitations, however, do exactly that.
Lack of User Validation on Rate My Professor
RMP does exactly what you’d expect: it allows students to post whatever they want about whomever they want. And this is the primary problem with RMP: that anyone can post anything.
I find my official ASU evaluations to be much a more helpful gauge of my teaching performance. No, they’re not perfect, but at least they students filling out the surveys actually registered for my class. (Whether or not they chose to attend, though, is another matter.)
Is it possible that there’s some valuable data hidden in my RMP profile—data that could help me improve my teaching chops? Sure, I imagine that there’s a high noise-to-signal ratio. Even if all RMP reviews came from my former students, there’s another and perhaps larger problem at play.
Bimodal Distribution of Reviews
Last semester two of my students scraped a bunch of data from RMP for their individual projects in my analytics class. They generally found what I suspected: Professor feedback tends to be bimodal. This is a fancy way of saying that students who post on the site tend to either love or hate their professors:
This is akin to many legitimate Yelp business and Amazon product reviews. For instance, let’s say that you went to a local restaurant. The service and food were fine. Would you take the time to write a review? I suspect that few would. However, if both were awful or outstanding, you’d be more inclined to spend the time on the app or site penning your thoughts. Invariably, this type of behavior leads to an availability bias.
While we’re at it, you can see the same phenomenon in many WordPress plug-in reviews. Here’s the histogram for Jetpack.
Trolling and the Data-Quality Dilemma
Certain professors even advocate gaming the system.
After college, I worked in SONY’s customer service department. Back then, my manager told me that angry customers told on average roughly 60 people of their experience. That number may be fifty times higher today. In some cases, it can be higher by orders of magnitude.
I don’t know how the numbers for irritated students compare, but I’d be astonished if at least some these folks don’t one or more of following unscrupulous things:
enlist their friends to write negative reviews about their nemeses
use sites such as MTurk and Fiverr to buy such reviews
register for more than one account on the site to post multiple (negative) reviews
Knowing this, professors can respond and counteract negative sentiment on RMP to some extent. Some even advocate gaming the system altogether:
Click to embiggen.
This means that data quality on RMP is highly suspect if not absent altogether—and I don’t see that changing under the company’s business model. RMP charges its users nothing. It relies upon ads for revenue. Make it too tough for students to post reviews (legitimate or not) and they may go elsewhere. More users mean more reviews and content. That means more ads served—ethics be damned.
Simon Says
No one likes to read negative press, but thoughtful ones can help. I can point to certain book reviews that stuck with me over the years. If you didn’t enjoy my class, though, then go ahead and bash away. I won’t be reading your RMP comments. You’re better off talking to me about your particular issue.
Feedback
What say you?
The post Why I Have Never Visited My Rate My Professor Page appeared first on Phil Simon.
