A defense of student evaluations: They’re biased, misleading, and extremely useful.
The answer requires us to think about power. If you look hard at the structure of academia, you will see a lot of teachers who, in one way or another, lack power: adjuncts and term hires (a large population, and growing); untenured faculty (especially in universities like mine); faculty, even tenured faculty, in schools where budget cuts loom; graduate students, always and everywhere. You might see evaluations as instruments by which students, or administrators, exercise power over those vulnerable employees. But if you are a student, and especially if you are a student who cares what grades you get or who needs recommendations, then teachers, for you—even adjuncts and graduate teaching assistants—hold power.
Chairmen and deans also need to know when classroom teaching fails: when a professor makes catastrophically wrong assumptions as to what students already know, for example, or when students find a professor incomprehensible thanks to her thick Scottish accent. My note: indeed, when chairmen and deans KNOW what they are doing and are NOT using evaluations for their own power.
Student Course Evaluations Get An ‘F’ : NPR Ed : NPR
Philip Stark is the chairman of the statistics department at the University of California, Berkeley. “I’ve been teaching at Berkeley since 1988, and the reliance on teaching evaluations has always bothered me,” he says.
Stark is the co-author of “An Evaluation of Course Evaluations,” a new paper that explains some of the reasons why.
Michele Pellizzari, an economics professor at the University of Geneva in Switzerland, has a more serious claim: that course evaluations may in fact measure, and thus motivate, the opposite of good teaching. Here’s what he found. The better the professors were, as measured by their students’ grades in later classes, the lower their ratings from students.
“Show me your stuff,” Stark says. “Syllabi, handouts, exams, video recordings of class, samples of students’ work. Let me know how your students do when they graduate. That seems like a much more holistic appraisal than simply asking students what they think.”
Do student evaluations measure teaching effectiveness?Manager’s Choice
Mauricio Vasquez, Ph.D.Assistant Professor in MISTop Contributor
Higher Education institutions use course evaluations for a variety of purposes. They factor in retention analysis for adjuncts, tenure approval or rejection for full-time professors, even in salary bonuses and raises. But, are the results of course evaluations an objective measure of high quality scholarship in the classroom?
Dr. Pedro L.
Learning analytics, student satisfaction, and student performance at the UK Open University
Rienties and his team linked 151 modules (courses) and 111,256 students with students’ behaviour, satisfaction and performance at the Open University UK, using multiple regression models.
There is little correlation between student course evaluations and student performance
The design of the course matters
Student feedback on the quality of a course is really important but it is more useful as a conversation between students and instructors/designers than as a quantitative ranking of the quality of a course. In fact using learner satisfaction as a way to rank teaching is highly misleading. Learner satisfaction encompasses a very wide range of factors as well as the teaching of a particular course.
this research provides quantitative evidence of the importance of learning design in online and distance teaching. Good design leads to better learning outcomes. We need a shift in the power balance between university and college subject experts and learning designers resulting in the latter being treated as at least equals in the teaching process.
more on learning analytics in this IMS blog
Weakest students more likely to take online college classes but do worse in them
Protopsalt is is a professor at George Mason University, where he directs Center for Education Policy and Evaluation. He previously served as a senior official in the U.S. Department of Education.
The paper, “Does Online Education Live Up to Its Promise? A Look at the Evidence and Implications for Federal Policy,” was also written by Sandy Baum, an economist at the Urban Institute, a nonprofit research organization.
At four-year universities, students with high grades often did just as well in an online course, but those with low grades suffered more. Another 2017 study of students at a for-profit university which offers both in-person and online classes found that students who took an online class not only got lower grades in that class but also in future classes. Online students were more likely to drop out of college altogether than similar students who attended in-person classes.
The question is whether we should keep expanding online learning, with generous federal subsidies, to the most vulnerable students before colleges have tested and proven they can educate them adequately outside the classroom.
more on online learning in this IMS blog
Giving Classroom Experiences (Like VR) More … Dimension
at a session on the umbrella concept of “mixed reality” (abbreviated XR) here Thursday, attendees had some questions for the panel’s VR/AR/XR evangelists: Can these tools help students learn? Can institutions with limited budgets pull off ambitious projects? Can skeptical faculty members be convinced to experiment with unfamiliar technology?
All four — one each from Florida International University, Hamilton College, Syracuse University and Yale University — have just finished the first year of a joint research project commissioned by Educause and sponsored by Hewlett-Packard to investigate the potential for immersive technology to supplement and even transform classroom experiences.
Campus of the Future” report, written by Jeffrey Pomerantz
Yale has landed on a “hub model” for project development — instructors propose projects and partner with students with technological capabilities to tap into a centralized pool of equipment and funding. (My note: this is what I suggest in my Chapter 2 of Arnheim, Eliot & Rose (2012) Lib Guides)
Several panelists said they had already been getting started on mixed reality initiatives prior to the infusion of support from Educause and HP, which helped them settle on a direction
While 3-D printing might seem to lend itself more naturally to the hard sciences, Yale’s humanities departments have cottoned to the technology as a portal to answering tough philosophical questions.
institutions would be better served forgoing an early investment in hardware and instead gravitating toward free online products like Unity, Organon and You by Sharecare, all of which allow users to create 3-D experiences from their desktop computers.
Campus of the Future” report, written by Jeffrey Pomerantz
XR technologies encompassing 3D simulations, modeling, and production.
This project sought to identify
- current innovative uses of these 3D technologies,
- how these uses are currently impacting teaching and learning, and
- what this information can tell us about possible future uses for these technologies in higher education.
p. 5 Extended reality (XR) technologies, which encompass virtual reality (VR) and augmented reality (AR), are already having a dramatic impact on pedagogy in higher education. XR is a general term that covers a wide range of technologies along a continuum, with the real world at one end and fully immersive simulations at the other.
p. 6The Campus of the Future project was an exploratory evaluation of 3D technologies for instruction and research in higher education: VR, AR, 3D scanning, and 3D printing. The project sought to identify interesting and novel uses of 3D technology
p. 7 HP would provide the hardware, and EDUCAUSE would provide the methodological expertise to conduct an evaluation research project investigating the potential uses of 3D technologies in higher education learning and research.
The institutions that participated in the Campus of the Future project were selected because they were already on the cutting edge of integrating 3D technology into pedagogy. These institutions were therefore not representative, nor were they intended to be representative, of the state of higher education in the United States. These institutions were selected precisely because they already had a set of use cases for 3D technology available for study
p. 9 At some institutions, the group participating in the project was an academic unit (e.g., the Newhouse School of Communications at Syracuse University; the Graduate School of Education at Harvard University). At these institutions, the 3D technology provided by HP was deployed for use more or less exclusively by students and faculty affiliated with the particular academic unit.
p. 10 definitions
there is not universal agreement on the definitions of these
terms or on the scope of these technologies. Also, all of these technologies
currently exist in an active marketplace and, as in many rapidly changing markets, there is a tendency for companies to invent neologisms around 3D technology.
A 3D scanner is not a single device but rather a combination of hardware and
software. There are generally two pieces of hardware: a laser scanner and a digital
camera. The laser scanner bounces laser beams off the surface of an object to
determine its shape and contours.
p. 11 definitions
Virtual reality means that the wearer is completely immersed in a computer
simulation. Several types of VR headsets are currently available, but all involve
a lightweight helmet with a display in front of the eyes (see figure 2). In some
cases, this display may simply be a smartphone (e.g., Google Cardboard); in other
cases, two displays—one for each eye—are integrated into the headset (e.g., HTC
Vive). Most commercially available VR rigs also include handheld controllers
that enable the user to interact with the simulation by moving the controllers
in space and clicking on finger triggers or buttons.
p. 12 definitions
Augmented reality provides an “overlay” of some type over the real world through
the use of a headset or even a smartphone.
In an active technology marketplace, there is a tendency for new terms to be
invented rapidly and for existing terms to be used loosely. This is currently
happening in the VR and AR market space. The HP VR rig and the HTC Vive
unit are marketed as being immersive, meaning that the user is fully immersed in
a simulation—virtual reality. Many currently available AR headsets, however, are
marketed not as AR but rather as MR (mixed reality). These MR headsets have a
display in front of the eyes as well as a pair of front-mounted cameras; they are
therefore capable of supporting both VR and AR functionality.
p. 13 Implementation
Technical issues can generally be divided into two broad categories: hardware
problems and software problems. There is, of course, a common third category:
p. 15 the technology learning curve
The well-known diffusion of innovations theoretical framework articulates five
adopter categories: innovators, early adopters, early majority, late majority, and
laggards. Everett M. Rogers, Diffusion of Innovations, 5th ed. (New York: Simon and Schuster, 2003).
It is also likely that staff in the campus IT unit or center for teaching and learning already know who (at least some of) these individuals are, since such faculty members are likely to already have had contact with these campus units.
Students may of course also be innovators and early adopters, and in fact
several participating institutions found that some of the most creative uses of 3D technology arose from student projects
p. 30 Zeynep Tufekci, in her book Twitter and Tear Gas
definition: There is no necessary distinction between AR and VR; indeed, much research
on the subject is based on a conception of a “virtuality continuum” from entirely
real to entirely virtual, where AR lies somewhere between those ends of the
spectrum. Paul Milgram and Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays,” IEICE Transactions on Information Systems, vol. E77-D, no. 12 (1994); Steve Mann, “Through the Glass, Lightly,” IEEE Technology and Society Magazine 31, no. 3 (2012): 10–14.
For the future of 3D technology in higher education to be realized, that
technology must become as much a part of higher education as any technology:
the learning management system (LMS), the projector, the classroom. New
technologies and practices generally enter institutions of higher education as
initiatives. Several active learning classroom initiatives are currently under
way,36 for example, as well as a multi-institution open educational resources
(OER) degree initiative.37
p. 32 Storytelling
Some scholars have argued that all human communication
is based on storytelling;41 certainly advertisers have long recognized that
storytelling makes for effective persuasion,42 and a growing body of research
shows that narrative is effective for teaching even topics that are not generally
thought of as having a natural story, for example, in the sciences.43
p. 33 accessibility
The experience of Gallaudet University highlights one of the most important
areas for development in 3D technology: accessibility for users with disabilities.
p. 34 instructional design
For that to be the case, 3D technologies must be incorporated into the
instructional design process for building and redesigning courses. And for that
to be the case, it is necessary for faculty and instructional designers to be familiar
with the capabilities of 3D technologies. And for that to be the case, it may
not be necessary but would certainly be helpful for instructional designers to
collaborate closely with the staff in campus IT units who support and maintain
Every institution of higher
education has a slightly different organizational structure, of course, but these
two campus units are often siloed. This siloing may lead to considerable friction
in conducting the most basic organizational tasks, such as setting up meetings
and apportioning responsibilities for shared tasks. Nevertheless, IT units and
centers for teaching and learning are almost compelled to collaborate in order
to support faculty who want to integrate 3D technology into their teaching. It
is necessary to bring the instructional design expertise of a center for teaching
and learning to bear on integrating 3D technology into an instructor’s teaching (My note: and where does this place SCSU?) Therefore,
one of the most critical areas in which IT units and centers for teaching and
learning can collaborate is in assisting instructors to develop this integration
and to develop learning objects that use 3D technology. p. 35 For 3D technology to really gain traction in higher education, it will need to be easier for instructors to deploy without such a large support team.
p. 35 Sites such as Thingiverse, Sketchfab, and Google Poly are libraries of freely
available, user-created 3D models.
ClassVR is a tool that enables the simultaneous delivery of a simulation to
multiple headsets, though the simulation itself may still be single-user.
p. 37 data management:
An institutional repository is a collection of an institution’s intellectual output, often consisting of preprint journal articles and conference papers and the data sets behind them.49 An
institutional repository is often maintained by either the library or a partnership
between the library and the campus IT unit. An institutional repository therefore has the advantage of the long-term curatorial approach of librarianship combined with the systematic backup management of the IT unit. (My note: leaves me wonder where does this put SCSU)
Sharing data sets is critical for collaboration and increasingly the default for
scholarship. Data is as much a product of scholarship as publications, and there
is a growing sentiment among scholars that it should therefore be made public.50
more on VR in this IMS blog
A Game, a Video, and a Framework for Teaching Website Evaluation
In this age of fake and misleading news being spread through social media, it is more important than ever to teach students how to view websites with a critical eye. Here are three good resources
The RADCAB website offers short explanations of each of the aspects of evaluation and why they are significant. The site also provides a rubric (link opens PDF) that you can download and print for your students to use to score the credibility of a website.
Website Evaluation Explained by Common Craft teaches viewers to think like an editor when reviewing the claims made in articles on websites.
more on fake news in this IMS blog
Student Entitlement: Key Questions and Short Answers
By: Maryellen Weimer, PhD
What is student entitlement?
“tendency to possess an expectation of academic success without taking personal responsibility for achieving that success.”
How widespread is it?
The research (and there’s not a lot) reports finding less student entitlement than faculty do.
Can a student be entitled without being rude and disruptive?
Yes. Students can have beliefs like those mentioned above and only discuss them with other students or not discuss them at all. Part of what makes entitlement challenging for teachers are those students who do verbally express the attitudes, often aggressively.
Are millennial students more entitled than previous generations? That’s another widely held assumption in the academic community, but support from research is indirect and inconsistent.
Is entitlement something that only happens in the academic environment? No, it has been studied, written about, and observed in other contexts (like work environments
What’s causing it?
A number think it’s the result of previous educational experiences and/or grade inflation. Some blame technology that gives students greater access to teachers and the expectation of immediate responses. Fairly regularly, student evaluations are blamed for the anonymous power and control they give students. And finally, there’s the rise in consumerism that’s now associated with education. Students (and their parents) pay (usually a lot) for college and the sense that those tuition dollars entitle them to certain things, is generally not what teachers think education entitles learners to receive.
How should teachers respond?
It helps if teachers clarify their expectations with constructive positive language and even more importantly with discussions of the rationales on which those expectations rest. Teacher authority gets most students to follow the rules, but force doesn’t generally change attitudes and those are what need to be fixed in this case.
October 18 for Student Entitlement: Truth, Fiction, or Some of Both and stay tuned for more in-depth information and resources that we’ll make available in Faculty Focus Premium in subsequent weeks.
References: Elias, R. Z. (2017). Academic entitlement and its relationship to cheating ethics. Journal of Education for Business, 92 (4), 194-199.
Greenberger, E., et. al. (2008). Self-entitled college students: Contributions of personality, parenting and motivational factors. Journal of Youth Adolescence, 37, 1193-1204.
Responding to Disruptive Students
Negative attention doesn’t help difficult students change their ways, but teachers can alter classroom dynamics through this exercise.
Draw a map of your classroom, including doors, windows, desks, blackboards—all significant items and areas. I’m sure you’ve already got a clear idea of where the most challenging students usually sit. Now imagine teaching class on a regular day. Trace the paths you usually take across the room. Do you sometimes speed up for a particular reason?
Now put your breathing on the map. Are you conscious of the way you breathe during class? Use a new color and draw a wavy line on top of the lines and arrows you’ve already sketched. Does the wavy line look even, or have you drawn some chaotic or nervous zigzags? Could it be that you’ve sometimes forgotten to breathe?
Investing time in building physical and emotional familiarity with the learning environment, instead of nervously anticipating disruption, changes the educator’s perspective toward the whole class, their interaction with individual students, and their self-awareness. Negative attention stops being a solution—instead it is seen as a hindrance to the process of understanding students’ needs.
Tobin, T. J., Mandernach, B. J., & Taylor, A. H. (2015). Evaluating Online Teaching: Implementing Best Practices (1 edition). San Francisco, CA: Jossey-Bass.
- 5 measurable faculty competencies for on line teaching:
- attend to unique challenges of distance learning
- Be familiar with unique learning needs
- Achieve mastery of course content, structure , and organization
- Respond to student inquiries
- Provide detailed feedback
- Communicate effectively
- Promote a safe learning environment
- Monitor student progress
- Communicate course goals
- Provide evidence of teaching presence.
Best practices include:
- Making interactions challenging yet supportive for students
- Asking learners to be active participants in the learning process
- Acknowledging variety on the ways that students learn best
- Providing timely and constructive feedback
- Instructor knowledge
- Method of instruction
- Instructor-student rapport
- Teaching behaviors
- Enthusiastic teaching
- Concern for teaching
8. The American Association for higher Education 9 principle4s of Good practice for assessing student learning from 1996 hold equally in the F2F and online environments:
the assessment of student learning beings with educational values
assessment is most effective when it reflects an understanding of learning as multidimensional, integrated and revealed in performance over time
assessment works best when the programs it seeks to improve have clear, explicitly stated purposes.
Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes.
Assessment works best when it is ongoing, not episodic
Assessment fosters wider improvement when representatives from across the educational community are involved
Assessment makes a difference when it begins with issues of use and illumines questions that people really care bout
Assessment is most likely to lead to improvements when it is part of the large set of conditions that promote change.
Through assessment, educators meet responsibilities to students and to the public.
9 most of the online teaching evaluation instruments in use today are created to evaluate content design rather than teaching practices.
29 stakeholders for the evaluation of online teaching
- faculty members with online teaching experience
- campus faculty members as a means of establishing equitable evaluation across modes of teaching
- contingent faculty members teaching online
- department or college administrators
- members of faculty unions or representative governing organizations
- administrative support specialists
- distance learning administrators
- technology specialists
- LMS administrators
- Faculty development and training specialists
- Institutional assessment and effectiveness specialists
Sample student rating q/s
Rate the effectiveness of the online library for locationg course materials
Based on your experience,
148. Checklist for Online Interactive Learning COIL
150. Quality Online Course Initiative QOCI
151 QM Rubric
154 The Online Insturctor Evaluation System OIES
163 Data Analytics: moving beyond student learning
- # of announcments posted per module
- # of contributions to the asynchronous discussion boards
- Quality of the contributions
- Timeliness of posting student grades
- Timelines of student feedback
- Quality of instructional supplements
- Quality of feedback on student work
- Frequency of logins
- 180 understanding big data
- factor structure
187 a holistics valuation plan should include both formative evaluation, in which observations and rating are undertaken with the purposes of improving teaching and learning, and summative evaluation, in which observation and ratings are used in order to make personnel decisions, such as granting promotion and tenure, remediation, and asking contingent faculty to teach again.
195 separating teaching behaviors from content design
more on online teaching in this IMS blog
The Online Discussion group for Blended and Online Learning leads an interesting discussion on course evaluations; here are the highlights:
When we first started in 1999, we included ~10 questions in addition to our standard questions that were different for online courses. This information was particularly useful as we grew our online offerings (i.e. Would you take another online course. 93-5% answered yes consistently. How would you rate the level of interactivity between you and the instructor? Between you and the other students?) These were administered via SurveyMonkey because there were no online evaluation services back then.
Now we have a single evaluation that is administered to all students regardless of the delivery format (online, hybrid, blended, F2F or intensive) The questions were designed to be relevant regardless of the delivery format. All of these evaluations are administered online…which has its downsides (e.g. response rate is less especially compared to what was captured in F2F classes in the past.) We continue to explore ways to increase the response rate.
Reta Chaffee Director of Educational Technology-Academic Affairs Granite State College 25 Hall Street Concord, NH 03301 (603) 513-1350
On Behalf Of Krajewski, Scott
Sent: Friday, February 17, 2017 1:00 PM
Subject: Re: [BLEND-ONLINE] Course Evaluations
Hi Hala, We have a standard online evaluation form for all courses. We do add 3 questions to the sports courses but otherwise we’re 100% standardized. We have a ton of info at
You might find this study (or the related literature) helpful — http://patricklowenthal.com/publications/Student-Perceptions-of-Online-Learning–Analysis-of-Online-Course-Evaluations.pdf
Patrick Patrick R. Lowenthal | Associate Professor Educational Technology, Boise State University http://www.patricklowenthal.com
On Behalf Of Rob Gibson
Sent: Friday, February 17, 2017 8:39 AM
Subject: Re: [BLEND-ONLINE] Course Evaluations
We use the IDEA evaluation framework combined with CampusLabs as the delivery engine.
- IDEA is a well-established evaluation process dating back to the 1970s.
- The CampusLabs delivery process (new as of about 2 years ago) provides students with a single URL to complete their evaluations – on-campus or on-line. Mobile friendly.
- It uses the same base evaluation criteria across the university. (That’s how IDEA is able to substantiate reliability and validity.) IDEA is matched against a national database using a CIP code. Hence, faculty can gather comparative data of their course against other similar courses in the university, or at the national level.
- While each department uses the same basic framework, there are modification that can be made. For example, custom questions can be added to the eval (these fall outside the scope of the comparative data) and the learning objectives can be modified by course, department, school, college. We have one School that has custom learning objectives for each course in their program. Objectives are set using a 3 point Likert scale.
Very easy to set up a survey administration. Data is retrievable within 48 hours after close.
more on online learning in this IMS blog
Discussion on the EDUCAUSE Blended and Online Learning Group’s listserv
develop anonymous mid-course student evaluations allowing students to reflect on course and progress and informing instructor about what is working or not in the course.
– what is working well for you in the course?
– what is not working well for you in the course?
- What is helping you learn?
- What is hindering your learning?
- What suggestions do you have to make the course better for you, your peers, or the instructor?
Katie Linder Research Director Extended Campus, Oregon State University 4943 The Valley Library Corvallis, Oregon 97331 Phone 541-737-4629 | Fax 541-737-2734 Email: firstname.lastname@example.org
At the University of Illinois, we have been using Informal Early Feedback as a way to gauge information from our students to help improve the courses before the end. Here are a couple of links to our site. The first is the main page on what IEF is and the second is the question bank we offer to faculty. This is a starting point for them, then we meet with those who want to work on tweaking them for their specific needs.
* About IEF: https://citl.illinois.edu/citl-101/measurement-evaluation/teaching-evaluation/ief
* Question Bank: https://citl.illinois.edu/citl-101/measurement-evaluation/teaching-evaluation/ief/ief-question-bank
If you have any questions at all, don’t hesitate to ask.
Sol Roberts-Lieb Associate Director, Center for Innovation in Teaching and Learning Pedagogy Strategy Team and Industry Liaison UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
more on student evaluations in this IMS blog: