What Constitutes an Open-Book Exam in the Digital Age?

Several weeks ago, I shared here the syllabus for the undergraduate class I am teaching this semester at USC. As I noted, it is my first time doing a lecture hall class in some years and my first undergraduate class at USC, so it has been a learning process for all involved. I wanted to share with you a pedagogical challenge I've faced this term in part as an illustration of the kinds of transition higher education is undergoing as we try to absorb new media technologies and practices into our teaching.

It starts with the decisions we made about the course readings. We opted to put the scanned essays onto Blackboard, the classroom management tool which USC urges us to use, rather than having them printed out at a local copy shop. My hope was to save the students money and to also save trees by having as close to a paperless class as possible.

Then, I made the announcement that the exams in the class would be open book, open note and that I was planning to distribute a list of potential questions in advance from which I would draw in constructing the exam, a practice I have used for more than 20 years without any great confusion. I've found that this approach lowers stress for students by allowing them to feel more in control as they are preparing for and taking the exam. In practice, some fraction of the class works really hard, prepares for the exam by writing out their answers in advance, and copies them into the blue book. Another fraction studies their notes, comes in and improvises on the exam, or develops an outline in advance that they write from. And some fraction, for their own reasons, pays no attention to the advanced questions, doesn't study, and does really badly on the exam. The kicker is often the identification questions, which would be simple to answer by anyone with a textbook open in front of them, but nevertheless often end up unanswered or answered wrongly. The result is a grade distribution curve not very different from what I would have if I gave a closed book, closed notes exam -- but as I said, it lowers stress.

No sooner did I announce this policy than I got a question I've never been asked before. A student wondered whether open book, open note, meant open laptop. I needed time to reflect on this and said I would answer in the next class period. Actually, it took me a few to get back to them with a response. Given this was a class on technology and culture, I decided to use this as a teachable moment.

So, I started by breaking down the computer into two elements. First, there is the computer as a stand alone word processing machine. I certainly would have had no great objections to students using the computer to write their answers or even to access their materials. Indeed, as someone with painfully bad penmanship, I had been the first in my graduate program to take my quals on a computer the department provided. They made sure to give me a clean disc as I entered the room and I was allowed to take nothing else with me into the test.

But this was before the era of networked computing, which fundamentally changed the character of what a computer is. So, allowing students to use a laptop during an exam suddenly would allow students to access any information anywhere on the web and more significantly would allow students to trade information with each other throughout the test in ways which would be extremely difficult to monitor.

As I thought about it, the challenges of designing a meaningful test under those circumstances intrigued me. What would it mean to create an exam which could be taken not by individual students but by networked groups of students -- either the class as a whole or a specifically designated study group? Could we enfold ideas of collective intelligence into the design of tests? Could we create challenges which demonstrated their mastery of the material through the search strategies they deployed and the knowledge they produced together? In theory, such an exam holds promise as more and more jobs require the capacity to pool knowledge and collaborate with a team of others to solve complex problems, and learning how to mobilize expertise under these conditions should be a key goal of our educational process.

But, how would we deal with such an exam in the context of our current grading systems? After all, we still assume that grades measure individual performance and so if we gave group grades, that might prove unsatisfactory to everyone involved. Would students raised in a culture where grades based on individual performance know how to act fairly in a culture where grades were based on group performance?

After all, we know that on group projects, bright students are often treated unfairly, exploited by their classmates, who fail to do their fair share of the work, and who may, in fact, not be capable of contributing at the same level? Under such system, teachers have had to devise systems to measure individual contributions to the group, thus going back to personalized rather than collective grading? What would be involved in terms of time and technology in monitoring what each student contributed to the group's collective performance on the exam?

And of course, all of this assumes that all of my students do have laptops or can borrow laptops, a more or less safe assumption given the relatively affluent population of USC, but hardly the case at many other colleges and universities around the country. How could you give one group of students such an intense advantage on the exam? Would we then have to issue laptops the way we now issue blue books?

As I started to contemplate these issues, I started to choke. As much as I wanted to be the cool, open-minded teacher, the model pedagogue for the digital age, there was no way I was going to be able to work through all of the implications of this radical shift in classroom practice in time to apply it this semester. A real answer to this question may not be possible in our current educational system, though it is a kind of question which we are going to be asked more and more. So, I spelled all of this out to my students, and challenged them to start thinking through the issues.

But, then came the turn of the knife. If they could not use their laptops, and the course texts had navigated to the web, then in what sense was this going to be an open book test? They could no longer access the course materials without printing them off, which would undo everything we saved by making them digital in the first place. The answer of course is that with the questions in advance, they could print out notes or print out the essays they needed to address the questions. They wouldn't have to print out everything, but they would no longer be working in a paperless environment.

So, we went back to the drawing board one last time, and asked the tech people if it would be possible to shut down the wireless in the room for the duration of the exams. They were not able/willing to do this, so that's where things stand as of the moment. Neither the students nor I are fully satisfied with this resolution, but both the pedagogical and technological structures of the modern university would seem to block any path out of this challenge that I have come up with.

I can't be the only faculty member on the planet facing these challenges, so I am posting this to see how other educators are dealing with these transitions. I can see the world we are surely evolving towards, but I don't know how to get there on my own.

So, let's use our laptops to work through this problem together. Oh, wait....

While we are on the subject of Digital Media and Learning, I wanted to give people a head's up for a great new documentary, New Learners of the 21st Century, which will be airing on PBS stations across the United States this coming Sunday, Feb. 13. Some of you will recall how one-sided and negative I found the Digital Nation documentary which aired last year, despite having talked to many key researchers and collected some compelling material for their webpage.

New Learners of the 21st Century offers the flip side of that documentary, taking us into innovative school and after school programs which are making creative use of new media platforms and practices for pedagogy. You can get a taste for what to expect from this opening segment which they have posted to PBS Video, but it is really, in this case, only the beginning.

By the second segment on Quest to Learn, the New York charter school which uses game design to teach, you can see the difference in the ways the two documentaries approach their topics. In Digital Nation, the Quest to Learn segment is almost incomprehensible: we see lots of activities involving technology but we have no idea what the kids are doing or why, and as a result, it feels like technology for technology's sake. Here, we learn about their pedagogical approach; we see processes unfold; we hear about when they use technology and when they ask the kids to put it aside. The focus is less on the use of computers in the classroom, an old topic after all and as my above discussion suggests, one we are still struggling with, and more on the use of new media literacies in education.

The same holds true for the film's treatment of a range of other pedagogical sites, including great stuff on work being done by the Smithsonian Institute and by the YouMedia Center at the Chicago Public Library, both important innovators in this space.

Because the topic is more narrowly focused, and because the goal is to explain and not simply stir up controversy, this film does do justice to the complex research which the MacArthur Digital Media and Learning program has funded in this area. I have been honored to be part of this initiative from the start, so my recommendation is scarcely unbiased here. But if like me, you've been burnt several times already by PBS's treatment of youth and digital media, I want to let you know that this one will be more rewarding.

Posted by Henry Jenkins at 7:30 AM

Comments