“We’re unlikely to have much luck improving student learning outcomes with technology if we don’t measure the impact of our technology investments and interventions.” Writes EdTechTeacher Co-Director, Justin Reich, as he addresses the essential question How Do You Assess Change? in his latest post EdTechResearcher.
In the last year or so, I’ve been thinking a lot about how I can be of service to school leaders who are trying to figure out how technology should fit in their schools. In general, I’ve been sharing two messages with school leaders: 1) technology should be in the service of defined learning goals and 2) schools and districts should assess whether technology is helping students make progress to those goals.
I have a particular soft spot for assessment because for the last four years I have run a research projectmeasuring the degree to which U.S. K-12 wikis provide opportunities for students to develop 21st century skills. But I also think assessment has several virtues that make it a useful entry point for getting school leaders to think more deeply about the challenges of tech integration in schools. First, as soon as you bring up assessment, people realize that they need to have some learning goals, or there is nothing to assess. Talking about assessment is one entry point to thinking about backwards design in schools. Second, the best outcome of No Child Left Behind (and I don’t have much nice to say about NCLB) is that it has raised awareness of the importance of formative assessment for learning. As schools as institutions learn about using technology, they need to evaluate their progress. Finally, many school and district leaders realize that assessment data can play a role in garnering support from school boards, voters, and trustees for technology investment.
So while researchers like me should be studying technology at state and national levels, every school and district should be studying their own technology use and figuring out whether or not “it’s working”–whether or not it’s meeting learning goals that educators and students care about. I’ve been telling school leaders that this kind of research doesn’t have to be ready for peer-reviewed scholarly journals–it just needs to be good enough to guide decision making and practice. Many educators hear “research” and “assessment” and they think that it needs to be complex statistical models and sophisticated surveys and rubrics. But we can’t let the perfect be the enemy of the good here. Better to do a few small things–some content analysis, some focus groups, a short survey– then to do nothing at all.
And I’m please to report that I have found an exemplar of the kind of educator action-research that can play a powerful role in shaping technology practice. I’ve long been a big fan of the thoughts and musings of Bud Hunt, AKA BudTheTeacher, since we were on an ISTE panel together a few years ago. He’s recently completed his Masters Degree with a research thesis, “Wait, Am I Blogging?”: An Examination of School-Sponsored Online Writing Spaces, that examines the use of blogs in his school district.It’s a fantastic piece of work, and great reading for anyone interested in assessing technology projects in schools. Bud’s thesis is posted on his blog here.
What did Bud do?
- He took a sample of all of his school’s blog posts
- He read them
- He categorized them
- He drew some conclusions about the kinds of learning taking place based on that analysis
In simplifying Bud’s thesis, I don’t want to minimize the work that Bud did. But I do want to highlight that it wasn’t rocket science; it was the kind of action research project that any team of educators in a school or district could tackle. His work is systematic, not complex, and his results are compelling and troubling. Basically, Bud found that most blogging activity was either teacher-centered content delivery or the kind of simple hub-and-spoke interactions typically found in classrooms, where a teacher asks a question, a kid responds, and a teacher evaluates. in general, Bud didn’t find young people finding a voice, developing an identity as a blogger, and pursuing topics and conversations with passion and interest. He found them producing perfunctory answers to the same kinds of questions they got in class. His research raises some important questions for the teachers in his district about blogging practice specifically and technology mediated collaborative learning more broadly. As Bud tweeted to me: ” My professional emphasis for the last two years has been on creating a culture of teacher research within the context of technology use and adoption. Hard. But worth doing.”
There is so much here that Bud does right in his research project that makes it a model for other educators. First, Bud doesn’t try to read everything that has ever been produced on blogs in his district; he takes a sample. He then investigates that sample in a systematic, but not overly complicated way. Basically, he looked at a pilot set of blog posts and developed a taxonomy of about 10 “purposes” of a blog post, and then he measures the distribution of those purposes in the full sample. In his analysis, he’s also fearless in confronting what his findings suggest about student learning opportunities. He found that blogging online looked too much like the kinds of writing that students did offline, and he challenges his colleagues to embrace the possibilities of the new medium.
It’s a tough message that Bud has to deliver to his fellow educators in his district. But it’s very difficult to imagine that community of educators getting any better without facing up to the analysis that Bud has done. (And I should add here that I think Bud’s findings square with my own assessment of the use of wikis in K-12 settings across the U.S.) This kind of assessment data gives teachers something concrete to use as they build a conversation about how they can get closer to their idea uses of technology.
So if you haven’t already, go read Bud’s thesis. Then if you are interested, come back and read the rest of this post. I have a two suggestions of things that Bud could have done differently.
Visit EdTechResearcher to read the more of Justin’s article.