Evaluating online teaching

17 Sep 2020 by Paul Schacht, CDL Director

Design Museum, London by Flickr user diamond geezer, CC BY-NC-ND 2.0

As a result of the coronavirus pandemic, many more Geneseo faculty are teaching online. Not surprisingly, then, department chairs, deans, and personnel committees are wondering how they can best evaluate their colleagues’ online instruction for the purposes of professional development and review.

Procedures and standards for peer review of faculty are, of course, properly a matter for shared governance and contractual negotiation. At some point it will fall to Geneseo’s college senate to approve guidelines for “observing” online instruction similar to those it adopted in 2006 for on-ground instruction, possibly together with some online-specific modifications to the personnel evaluation forms for tenured/tenure-track and adjunct faculty. But because schools and departments need help with this question right now, I put down some thoughts and shared them with the CDL faculty affiliates, student affiliates, associates, and leadership team, as well as the Academic Experience Planning Team convened last spring by Provost Robertson to develop recommendations for the transition to remote learning. I also consulted Wes Kennison, Geneseo UUP chapter president. I got a lot of terrific feedback, especially in the faculty affiliates’ meeting on September 9.

What follows isn’t a group document but rather my best advice, informed by the range of perspectives I heard in that feedback. As you might expect, the perspectives didn’t line up neatly, but there was significant disagreement on only one point, which I discuss below.

I’m especially grateful to Wes for his precise insistence on the difference between mentorship and personnel review as processes. While the advice below raised no red flags for him, he cautioned that his personal lack of objection should not be construed as an official union endorsement.

Tl;dr edition

For those who like their conclusions first, here’s the short version of my advice:

• Be guided by best practice for evaluating online instruction, but adapt it to present circumstances: faculty workloads and pressures right now call for a streamlined approach that respects everyone’s limited time as well as the psychological strains on colleagues under review, especially those without tenure.
• In establishing a temporary procedure for your school or department, consider the difference between physical and online spaces when it comes to both faculty’s and students’ sense of privacy and safety. Various online equivalents to the conventional collegial classroom visit may leave some people feeling uncomfortably exposed or surveilled and could cause some students to feel as though their trust has been violated.
• Whatever approach your school or department takes, make sure everyone fully understands the why and how of it and that everyone follows it consistently.

Two questions, not one

For Geneseo right now, the question of how best to evaluate online teaching is really two questions. First, in the abstract, what constitutes best practice for peer evaluation and mentorship in this area? But second, what’s a fair, reasonable, and practicable approach to assessing and mentoring colleagues under our present unusual circumstances, where many instructors, evaluators, and students are operating in unfamiliar territory, with new instructional modalities and with tools they may be trying for the first time?

Best practice calls for a thorough review of an instructor’s online course design and content in light of widely accepted, research-based standards. Many institutions use the adaptable, standards-driven review process developed by Quality Matters, a nonprofit organization that grew out of a higher-education consortium in Maryland. In collaboration with the Online Learning Consortium, SUNY has developed its own instrument for developing and assessing online courses, the Online Course Quality Review Rubric: OSCQR for short. Both tools have the advantage of being backed by extensive research on effective online instruction. They’re thorough and detailed: OSCQR, for example, comprises 50 standards arranged under six dimensions of online course architecture and implementation: Overview and Information, Technology and Tools, Design and Layout, Content and Activities, Interaction, Assessment and Feedback.

However, these tools are intended as frameworks for the full online course evaluation process, from development through review to improvement. They presume an instructor who’s deliberately chosen to teach online and to build courses informed by the tools’ design principles. They also presume evaluators thoroughly familiar with those principles.

Needless to say, that’s not the situation we’re in right now.

Realistic alternatives to best practice

Under present circumstances, a better approach to evaluating instructors’ courses would be to rate them against just a few, high-level principles of effective practice. One starting point might be Geneseo’s Course Readiness Checklist, the greatly streamlined version of OSCQR developed by CDL Assistant Director for Online Learning Laurie Fox and her team of instructional designers in CIT. Departments could adapt it to fit distinctive aspects of their disciplines or cultures.

Another possible starting point is the “Seven Principles for Good Practice in Undergraduate Education” distilled from research on student learning by Arthur Chickering and Zelda Gamson in their influential 1987 article of that name. Because these principles apply to every course modality, from fully in-person to varieties of hybrid to fully online, they’re likely to be recognized and understood by the largest number of faculty. It’s no accident that they track well with the indicators of effective “Contributions to Teaching” listed in the “Appendix” to Geneseo’s Personnel Evaluation Form (PER) as updated in May 2019.

The best undergraduate teaching, Chickering and Gamson argued,

• Encourages contacts between students and faculty
• Develops reciprocity and cooperation among students
• Uses active learning techniques
• Gives prompt feedback
• Emphasizes time on task
• Communicates high expectations
• Respects diverse talents and ways of learning

I’m not keen on the phrase “Emphasizes time on task,” but properly translated it simply means that good teaching communicates clear expectations for how students should apportion their time in realizing a course’s learning outcomes. This clarity is especially important in asynchronous online courses, where students bear responsibility for effectively managing the 2,000-plus minutes that an in-person three-credit course, for example, fixes in class meetings.

At Penn State College of Earth and Mineral Sciences, Ann Taylor, a member of the Dutton E-Education Institute, developed a Creative Commons-licensed Peer-Review Guide for Online Courses based on Chickering and Gamson’s seven principles. Taylor’s peer-review guide indicates where reviewers could look within an online course to see how the course measures up against these principles.

Geneseo deans, department chairs, and personnel committees might consider adapting Taylor’s guide to come up with a form and a process that reflects the particular ways in which their respective disciplines realize the seven principles; weaves in additional high-level Geneseo values such as equity, inclusion, and accessibility; and respects the constraints of time and experience under which we’re all operating right now.

Faculty undergoing personnel review typically prepare a self-reflective statement together with evidence of their teaching effectiveness that includes course materials (syllabi, handouts, assignments, etc.) and examples of student work. In writing reflectively about their online teaching, faculty might describe how they see the design and content of their online courses meeting the discipline-specific version of standards such as those described above. Additionally, as evidence of growth, they might explain how they’ve changed their online teaching in light of previous experience. And they might illustrate their effectiveness by including examples of student engagement such as de-identified excerpts from discussion forums.

Transparency, consistency, privacy

Whatever approach is taken, the important thing is to make sure the process and the standards it employs are transparent to faculty under review and followed consistently by reviewers. Taylor’s guide spells out how Penn State instructors should initiate and reviewers carry out their process. Academic departments at Geneseo should do the same.

However it works, the process will require instructors to share their course design and content with one or more peers. It’s best not to have them do this by simply granting peers access to the course in Canvas. Students who’ve contributed to the course’s online discussion forums, posted videos, and otherwise shared their views and ideas with the instructor or classmates weren’t expecting these outsiders to be part of their audience.

CIT can instead clone the course so that peers can see its structure and instructional content but no student-generated content. To give reviewers a feel for how they engage students in active learning, instructors can, as suggested above, include de-identified samples of student work as part of their self-reflection.

What about synchronous observation?

When evaluating physically present teaching, we traditionally attach great importance to classroom visits, which give us the opportunity to see how a colleague interacts with students. Online, there’s no classroom to visit, so course design and content become the most important indicators of quality — not only out of necessity, but because the absence of a classroom makes them that much more essential to effective instruction and student success.

If an online course has a synchronous component or is mainly synchronous, however, it’s possible for a colleague to “drop in” on a Zoom lecture or discussion in an attempt to get a feel for an instructor’s pedagogical style. Is this a good idea?

We’ve arrived at the one point on which the colleagues I consulted in preparing this blog post had significant disagreement, with a small number arguing that by visiting a live synchronous meeting or watching a recorded one, a peer reviewer might gain meaningful insight into a colleague’s effectiveness with minimal disruption to the class as a community.

My own advice is: don’t do it. I’m skeptical about the insight and very concerned about the impact on community — as a general matter, but particularly right now.

I can imagine a future — hopefully a pandemic-free one — where videoconferencing has become so commonplace and the technology has made it so intuitive that we can all read people’s conferencing interactions as easily as we do their in-person ones, a future where we can confidently assess the online equivalent of how well a colleague “uses the board” or facilitates a sense of community in a physical classroom. (How nimble is their screen-sharing? How effectively do they use breakout rooms? Are students genuinely engaged?) But until we get there, and considering that many faculty are working hard to gain fluency with a raft of new tools and methods for live online engagement, I worry that reviewers will reach unwarranted conclusions while generating tremendous stress for their colleagues.

I worry, too, that a visit to a synchronous online meeting may feel “creepy” to some students in a way that a visit to a physical classroom doesn’t, in part because online there’s little if any possibility of anonymity. Student experience here may be shaped in part by the sense that in general the educational sector is watching them constantly, using their phones to track their movements around campus, asking them to install spyware on their computers in an effort to combat academic dishonesty, or even sending information about their activity to companies like Facebook and Google. The point isn’t that these things are happening at Geneseo but rather that they form part of the climate in which students in general now learn. Meanwhile, one thing that is happening here — necessarily, and for the best of reasons — is intense daily monitoring of students’ health and social contacts. Under the circumstances, I think we’d do well to avoid, where possible, adding anything else to the atmosphere that might feel like surveillance.

Would asking a colleague to record a synchronous online meeting mitigate any of the potential harms of a live visit? I suppose it might reduce the colleague’s anxiety somewhat, but it wouldn’t remove the surveillance issue and might, in fact, exacerbate it by creating a permanent record of the things people said and did in the meeting, with identities attached, a record whose future circulation would lie outside any of the participants’ control. I think faculty have as much reason as students to be wary of such an outcome.

Here, again, then, my advice is: don’t do it.

Student opinion

It’s worth considering what role student opinion might play in evaluating online teaching. At Geneseo, our process for evaluating physically present teaching has for many years included student opinion through the SOFI survey. The college senate is currently reviewing this practice with an eye toward turning a radically revised SOFI into an instrument for formative assessment that would no longer play a role in personnel evaluation. Eventually, it may make sense to have a dedicated instrument for evaluating online teaching that would serve the same developmental purpose and would be similarly excluded from use in personnel decisions. It might ask the kinds of questions that could help a faculty member gauge how well they succeeded in, for example, establishing a personal presence, making students feel welcome and connected, tying activities and assignments to course learning outcomes, and communicating expectations and feedback effectively.

Whether and how student opinion should play a role in improving online teaching and learning at Geneseo is, of course, another matter to be determined through shared governance.

Some final thoughts

To repeat, what you’ve been reading here is advice, not policy. Hopefully it’s helpful advice at a moment when, without settled policy to guide us, it’s nevertheless important for professional development and personnel review processes to continue.

Throughout this post, I’ve spoken of those two processes in tandem because I’ve been focused on identifying the standards to which faculty should look both to become better online instructors and to demonstrate how they’re meeting expectations set by their departments and the college. While the standards may be the same, however, the processes are of course quite different, as are the stakes, particularly for faculty whose continued employment at a time of heightened precarity may hang in the balance.

In developing transparent and consistent standards adapted from best practice with an eye toward our unprecedentedly stressful and unsettled moment, then, departments and schools should also keep in mind that, from an ethical standpoint they should not, and from a contractual standpoint they may not, judge faculty’s performance by criteria of which those faculty were not previously informed. To put this more simply: look in Canvas for appropriate evidence that faculty are doing, in that environment, and to the best of their ability, the things that have been traditionally associated with excellent teaching at Geneseo.

As you look, remember how challenging it’s proving for many faculty to find their feet in a teaching environment where they’ve landed suddenly, and not by choice. Pay particular attention to the things they’re getting right.

Further reading

If you’d like to explore some of the resources mentioned here or additional ones that CDL faculty, students, and staff have been gathering on this topic and others related to digital learning, I invite you to follow our public bibliograpy on Zotero.