Some Study Results Might Not Be Relevant

I came across an article in The New York Times about how educational technology companies inflate success rates and withhold negative results of studies done to evaluate their software. It seems that in many cases, students don’t show significant improvement on standardized tests after using some software programs.

The article focused on one particular software program, Carnegie’s Cognitive Tutor. At a school district in Augusta Georgia, teachers noted that after using the program, students who had been performing below average made enough progress to catch up with students in mainstream classes. Classroom benefits, though, don’t always yield the same results on standardized state tests.

“On the other hand, when the new state math test was given in March, 27 percent of the district’s 11th graders did not pass. . .”

It might be that students don’t do so well on the state tests because they are used to the way the software program presents information. When they take the test, which in most cases is paper and pencil, the difference between reading and working out problems on a computer versus choosing the correct multiple choice answer and entering the answer on a Scantron bubble sheet might cause students to work slower to adjust to the change and consequently, answer fewer questions. The difference in presentation might also cause confusion, which might result in more wrong answers.

This doesn’t excuse educational software companies for cherry-picking data or omitting whole studies altogether. But it might explain why the results of some studies aren’t relevant.

Tutoring and Technology

I attended a Teaching and Learning with Technology Committee Meeting (I’m a co-chair for the Norfolk campus) on Tuesday, July 11. The main topic was the college’s technology plan. The committee had been tasked with drafting the instructional side of the plan. I brought up the topic of SMARTHINKING and the committee members discussed it at length. The committees main concern was the low number of contact hours per student.

 

Our interim Vice President of Distributed Teaching and Learning Services, Dr. Yanyan Yong, said she would ask faculty to recruit students to print out their sessions for a qualitative analysis of SMARTHINKING. I offered to print out the peer reviews that the tutors had done for my ENG 3 students that summer and Dr. Yong expressed interest in seeing those as well. Committee members also agreed that student feedback isimportant—a technology plan that doesn’t address student support services is inadequate.

A Stranger Comes to Town

It’s been about six months since I last wrote. While this is not an epistolary blog, I do feel as if I owe someone a letter. To continue the story of the writing center, the next four tutor training sessions went pretty much the same as the first four. Once I showed the tutors how to find their way around inside Blackboard, and showed them examples of peer reviews that Donald and I had done, they went to work. There were some ebbs and flows, with people leaving and returning from vacations and trips. It required some juggling, and I ended up taking up much of the slack. But it was all good. This was a test drive, after all.  

The big whoop dee do occurred in the sixth week of our research project. SMARTHINKING came to our campus, offering training sessions for faculty and staff who planned to use their tutoring service in the fall. The SMARTHINKING trainer demonstrated how SMARTHINKING works, showed us how to access SMARTHINKING services, and how to assist students with using it. Here’s basically how it works (my remarks are paraphrases, sorta, kinda, from SMARTHINKING’s website blurbs):

Live, Online Tutoring

Students can meet with SMARTHINKING e-structors in a chatroom for live, asynchronous help. Students can schedule a session ahead of time or they can just drop in. The e-structors use Whiteboard Technology to demonstrate concepts and answer questions.

Online Writing Lab

Students can email written assignments for a detailed critique from a SMARTHINKING e-structor. Students get feedback on the strengths and weaknesses of their assignments, as well as specific revision suggestions. Students may choose a 30-minute review or a 60-minute review for longer essays.

Essay Scoring

SMARTHINKING e-structors will review student essays and assign a score based on seven writing elements. Faculty can view their students’ essays and e-structor feedback. Instructors may use scores and comments to shape course assignments and feedback for student writers

Hosted Technology

SMARTHINKING licenses its technology platform to clients and provides training and consulting on the delivery of online tutoring programs. Schools can combine their tutors with SMARTHINKING’s online tutors to create a comprehensive online tutoring infrastructure.

SMARTHINKING did five sessions. About 20 people were at the session I attended. When I arrived at my session, Faith and Ellynne were already there. I sat near them and we talked before, during, and after the session (I am a bad, bad, bad student; I would hate to have someone like me in one of my classes). The overall consensus among the three of us was that having online tutoring available through SMARTHINKING was better than no online tutoring at all, which is what we had at the moment.

 

None of us was pleased with the writing tutoring session they showed us. It was a synchronous chat session, using a whiteboard as the medium. We thought the tutor in the writing demonstration spent too much time on a picky grammar rule and no time on higher order concerns. And the student had major higher order issues. It sounded like faculty would be able to view their students’ sessions, which is probably good. I would want to know if students were getting good suggestions. Students are allocated only two hours of tutoring per semester. If they need more hours, they will have to pay for it themselves. It’s $34.99 per hour but they would have to order a minimum of two hours, for $69.98. Few of our students can afford their prices. The college has said maybe we could work something out where students who needed more time could use hours from students who weren’t using the service. We’ll see.

Faith had to leave after awhile, but I stuck around and talked to Ellynne and another writing tutor. Later in the week, I talked to Etta. The tutors had a much stronger reaction to SMARTHINKING than I did. They hated it. It was too impersonal. They weren’t impressed by the tutoring session demos. They thought that two hours per student would not be enough time for students who need tutoring the most. They wanted to know what would happen to the hours students didn’t use. They were concerned about the cost (we all felt that the college should have spent the money on our own tutors and tutoring centers). I promised the tutors that I would bring this issue up at the next Teaching and Learning with Technology meeting on July 11, which I’ll talk about in my next installment.

Fourth Session: A Period of Adjustment

In preparation for the next round of peer reviews, I created groups within Bb for each tutor and their students on Monday. I did an analysis of the previous peer review forum and saw that many students did not get any peer reviews at all. Some got only one review (they were supposed to give and get two), and two students got more than seven reviews. Most students did the required two reviews. Some students did more than the requirement. A few didn’t do any reviews at all, which violates Jolemore’s Golden Peer Review Rule: “Give the same number of reviews unto others as ye would have them give unto you.” I made a point of relating this information to my students in my Monday announcements. I hoped that smaller groups would encourage students to give more reviews. They wouldn’t be able to hide their lack of participation in a smaller crowd.

 

On Wednesday, when I arrived at the writing center, things were pretty busy, so I worked one on one with each tutor. I explained the writing assignment for that week, showed them how the groups feature works, went over the peer reviews guidelines, and gave them the due dates for each phase of the drafting and revision process. I felt comfortable with their understanding of how everything worked in my class, now that they had gone through a full cycle of peer reviews. I was hoping that the following week, we would be able to get several peer reviews done without too much trouble.

Third Session: Dismay and Despair

During the week, my students had made a mess of the discussion threads. I had set up the peer review forum so that all they had to do to post their drafts was to open their tutors’ intro and hit reply. Instead, they replied to my post, or they created completely new threads, or they replied to their tutors, but changed the subject line so the tutors couldn’t tell where their drafts were. I thought about chewing them out and making them re-do their posts, but I figured they would only get more confused, I would have to answer dozens of questions, and I’d end up more frustrated than before. So I made a mental note to revise the peer review posting instructions (which I did) and I prepared myself to deal with the ensuing chaos.

I had planned to conduct a norming session on formative evaluations with the tutors to discuss ways they could respond to the students’ rough drafts. I wanted to make sure that the comments students received were consistent. I felt students should get the same quality of response no matter which tutor they were assigned to. I also thought a norming session would give us a chance to talk about our methods of responding. I printed out copies of a few sample papers at home, and drove through pounding rain and flooded streets to the downtown Norfolk campus.

My session with the tutors that day reflected some of the chaos I had witnessed among my students’ mangled use of the peer review forum. The situation was out of my control before I even got to the writing center. Only two tutors were there, Ellynne and Lee, and they had already posted their peer reviews, but hadn’t followed the format I outlined the week before. I struggled with my impulses about how to react. Should I try to impose more order and control over the situation—break bad on the tutors—or should I be more flexible and work with the flow? I opted for a gentle nudging toward a more coherent direction and mentally filed the experience under “lessons learned.”

We did talk about the chaos in the peer review forum (they were having trouble figuring out where their students’ drafts were). I told them I could create group areas in Blackboard. Groups have some advantages (less chaos, more control), but the drawbacks are that students can only read and respond to the people in their group. It cuts down on choice and it cuts down on opportunities to read more of their classmates’ drafts. I think students learn much from reading each other’s drafts, especially students who need models for a level of writing that’s just a notch above their own. Managing groups is also more work for me initially, because I would have to post the guidelines in each separate group, but it would probably be less work in the long run, because I wouldn’t have to spend so much time trying to figure out who was responding to whom.

Later, we talked about how to respond to the students’ rough drafts. Lee felt we should challenge them to expand their vocabulary by using words in our reviews that they would have to look up. I was in favor of using simpler words for assignments, instructions, and peer reviews— students should certainly be challenged by the texts they read, but when teachers and tutors communicate with students, I think our primary purpose is to be understood quickly and easily.

Lee and Ellynne both had questions about the peer review guidelines. The first three items in the guidelines address the introductory paragraph: Is there enough background info? Is the thesis clear? Does the writer’s purpose come through clearly? Lee and Ellynne suggested that the items about thesis and purpose could be combined because one affects the other. They also felt that background wasn’t relevant to the assignment, which is to write a letter to a child or parent explaining a value that they hold. I agreed that background may not matter for that assignment but the writing situation does. So I decided to revise the peer review guidelines for the next semester.

Second Session: Getting to Know You

The tutors had seemed especially interested in learning how to use Blackboard, so I started the second session with a Blackboard tutorial. I showed them how to login, how to navigate within the site, and how to use the discussion board. They posted short intros in the most recent forum, which was the peer review forum my students were working in at the time. I printed out the peer review guidelines so we could move to a worktable and talk more easily (the writing center computers face each other, which makes conversation difficult).  

I asked them about their writing center experiences: What kind of help do students most often come in for? How do they (the tutors) handle peer reviews? What methods, strategies, and techniques do they employ? I asked because I wanted to know what the tutors do when students come in for peer reviews—as well as what other kinds of help students ask for in the writing center. Even though my research project centers on responding to student writing online, I’m also interested in other tasks the tutors do with students and whether those tasks could be done online. I thought it may be possible to have an email Q and A service, but we would need to create a single writing center email account or maybe create an online writing center help form accessible from TCC’s website. 

The tutors told me that many students come in for a quick fix, or for a quiet place to work. Some students ask for help with accessing their email or their Blackboard site. Some need help with using Microsoft Word. Some students want a tutor to look over their draft quickly; some need help getting started or choosing a topic. At the end of the semester, they get many last minute procrastinators. Some students want the tutors to tell them what to do from start to finish. 

As for peer reviews, the tutors all agreed that they need to see the assignment from the teacher. If a student doesn’t bring one or if the teacher didn’t give students a hard copy, then the tutors still help them, but in a more general way. They read a student’s paper while the student sits and waits. Then they give their suggestions. They write their suggestions on a separate piece of paper, being careful not to write on the student’s paper. The tutors said they focus on different writing concerns; higher order content issues for college level writers and more on structure and grammar for developmental writers.  

We talked about the differences between face-to-face tutoring and tutoring online. I showed them some sample reviews done by me and my former instructional assistant, Donald. Ellynne remarked that Donald “talked” nicer to the students than I did. I admitted that my tone sounded more direct and I gave less praise, but I reasoned that it was from doing so many reviews, for such a long time. I won’t lie to students, I told them. I won’t say something is great when it’s not. But I could probably work on improving my tone. 

Before I left, I outlined the project schedule for the rest of the semester. I talked about how the cycle of drafting, peer reviewing, and revising goes for my students and about where we would fit our responses in—the due date for my students to post their first draft was the following Monday (and most students wait until the last day to post, of course). The following Friday, they were supposed to have given peer reviews to two classmates. Over the weekend, they were supposed to revise their drafts based on feedback from peers, tutors, and me. The final draft was due the next Monday.

First Session: Perceptions, Experiences, and Attitudes

We started the summer phase of the project with five tutors; Amber, Ellynne, Etta, Glynis, and Lee. Our first meeting was on Wednesday, May 31, about a week into the summer semester. My plan was to keep the training as open-ended as possible. I wanted to get feedback from the tutors themselves because they were the ones who would be on the front lines. I wanted to give them some control over the shape and direction of the project.

 

I began the first meeting by explaining what we were going to do, what my own perceptions, experiences, and attitudes were about tutoring online, and some of what I hoped to learn from our project. I then asked the tutors to fill out a survey about computer proficiency in a variety of areas so I could get an idea of what skills they had. Glynis, who later was diverted to the main tutoring center, was the most computer proficient, and had done some online tutoring for math via email. Ellynne was the next most proficient. She had been one of my students in English 112, and in that class, she had gained some experience with Blackboard. Lee and Amber’s computing skills were very good. They had also worked with Blackboard. Etta reported the least proficiency and expressed some anxiety about participating in the project.

 

I collected the surveys, and then I asked the tutors what they thought about the concept of online tutoring. Ellynne thought online tutoring would work well for college level students but not so well for developmental writers. I shared my own experiences with teaching developmental writers online. The tutors were surprised that English 3 students could do well in an online class. This is a common misperception even among English teachers, mainly because they are judging from their experiences with developmental writers in face-to-face classes. Online classes, however, attract many students who can’t take classes on campus. Most of them have adequate computing skills for getting around in a course management system. Students whose skills are not adequate usually drop the class before the drop/add deadline.

 

After we talked about our perceptions of the project, I gave the tutors some suggestions about how we could approach tutoring online. I told them about an experience from two semesters back, in which Donald, one of our former tutors, had been assigned to my on campus English 3 class as an instructional assistant. Donald had worked with my students in the computer lab (he has good skills) and in small group, classroom peer reviews until he got a good understanding of how my classes worked. The next semester, he worked with my online students, giving them peer reviews in the Blackboard discussion board. I told the tutors that the work I did with Donald had given me the idea to work on this project.

I said I thought that we could take turns doing revision peer reviews in the Discussion Board—one tutor per week. Ellynne felt that it would be best for the tutors to work throughout the semester with the same students so they could develop relationships with them. I agreed, and we decided to group the students in clusters and assign one tutor per cluster. Before I left, I showed the tutors around my Blackboard site, enrolled them all in the site as teaching assistants, and told them they could go in anytime to look around and get a feel for the Blackboard environment. I also printed out one of the forums so the tutors could see how students were responding to each other.

A Story about an Online English Tutoring Project

I haven’t posted anything to this blog in ages. My enthusiasm for reading blogs hasn’t transformed me into the prolific blogger I had hoped to become. However, the fact that I don’t write here doesn’t mean that I don’t write at all. I keep a journal for expressing my personal thoughts and I carry with me at all times another journal to write down my teaching experiences and impressions. Some of the material from my teaching journal has made its way into an article I’ve been working on for Inquiry, a Virginia Community College System publication for teachers in the VCCS. What follows today and for several days hereafter, are excerpts from my article in progress. 

Prologue: This is a story that chronicles the development of an idea for an online tutoring program. The final stage of development was a research project, “Online Tutoring for Online English,” conducted as part of a VCCS Professional Development Grant project during the spring and summer of 2006. 

In the summer of 2001, when I was still a part-time instructor at TCC, my dean asked if I were able and willing to teach a section of ENG 3 online. I hesitated at first—not because I wasn’t somewhat proficient with online technologies; I had been publishing course web sites since 1998 and using Blackboard for my on-campus classes since 2000. The truth was I didn’t believe ENG 3 students had the necessary skills to be successful in an online class. I based my belief on experience with my on-campus ENG 3 students in the classroom computer lab, where I would spend much time showing students basic computing tasks. I assumed that students in an online version of ENG 3 would be similarly challenged, but the chance to teach a course and receive a significant stipend convinced me to give it a try. Since it would be my only class that summer, I would have plenty of time to create additional course materials and work one on one with students.  

What I discovered that summer partly supported and partly disproved my assumptions. Students displayed a wide range of computing proficiency. Some students were skillful computer users while others struggled with the most basic computing tasks. I struggled along with my students to find ways to explain how to login to Blackboard, post to the discussion forums, send email, or check their grades. Students’ writing skills varied as well. Some students’ writing showed they were almost ready for ENG 111, while others grappled with serious grammar, spelling, and sentence structure problems. And I grappled for words to unlock their understanding of Standard English. 

After that first experience, I continued offering ENG 3 online. I took on another section, on top of the on-campus classes I was assigned, which meant that I wasn’t able to give my online students as much one-on-one help as I had that first semester. Teaching online involves almost twice as much work as on campus teaching, which of course, puts a greater burden on online teachers, which means they have less time and energy to give students individual assistance. I soon realized the need for some kind of tutoring service for my online students. In my on-campus classes, when I see students who need more help, I refer them to the tutoring center on campus, but many of my online students report that they can’t come to campus at all or can’t fit campus visits into their schedule. Many online students are active duty military, some of who take classes while overseas or on cruises. Some students are stay at home parents with no family or friends to watch their children while they’re at school. Some have full time day jobs. Some don’t live in the Hampton Roads area. Some students have told me about physical or psychological disabilities that make coming to campus difficult.  

Online courses also have a lower retention and success rate than traditional face-to-face classes. Students give various reasons for dropping or withdrawing from an online course. The reasons cited most frequently in an Online Student Survey conducted by Distributed Teaching and Learning Services in Spring 2006 were “changes in work schedule” (12.5%), “family obligations” (11.1%), and “more work than expected” (11.5%). What was missing from the survey was a question about tutoring support services. The survey asked about student satisfaction with Blackboard services, the Student Information System, student webmail services, and online learning resources and library materials, but not about tutoring. And not one of TCC’s four campuses offers tutoring services online. In the same Online Student Survey, 17.5% of students responding said that the most important reason they were taking online classes was that they could not attend class on campus.  

Why, I wondered, in my first year teaching online, didn’t we have the same academic support services for online students that we had for on-campus students? Over the next four years, my desire for an online tutoring service led me to investigate ways to adapt on-campus tutoring to an online environment. In fall 2002, I volunteered to be on a Tutoring Center Task Force and shared my thoughts with the other task force members. I searched for technologies to use. I went office to office making informal inquiries. I exchanged emails with faculty who expressed interest in online tutoring.  

Finally, in spring 2005, I had an opportunity to put some of my ideas into practice. I was assigned a supplemental instructional assistant for my on-campus ENG 3 class. Donald worked with my students in the computer lab and in small group, classroom peer reviews until he got a good understanding of how my classes worked. My students liked him and he was very helpful in the lab. The next semester, he wanted to cut back on his hours on campus, so I asked him if he would work with my online students, giving them peer reviews in the Blackboard discussion board. Donald did so well in my online class that I decided to work on a VCCS Professional Development proposal for a research project on online tutoring. That fall, I wrote a proposal for a VCCS Professional Development grant to collaborate with the Writing Center director, Faith Hutchinson, on developing an online tutoring program. 

Our original plan was to identify technology that would facilitate online tutoring for students in online writing classes, to evaluate technologies to determine which technology would work best, to determine the best formative assessment practices, to develop training procedures for tutors and to implement online tutoring at the Norfolk campus, and if successful, take it to TCC’s other three campuses. We were advised to extend the research over two semesters, spring and summer 2006. We decided to focus on technology and best practices research for the spring and planned to do tutor training in the summer. For the first half of the spring semester, Faith and I met often to compare notes on our project. About halfway through the spring semester, the college announced that TCC would be using SMARTHINKING in the fall. SMARTHINKING was supposed to be an alternative to our on-campus tutoring services for online students in the 2006-07 academic year. At the time the announcement was made, I thought our research project had suddenly become obsolete.  

 I talked to Faith about whether we should even continue working on our project. Why search for technology to allow students to receive tutoring help online from our college’s tutors when SMARTHINKING could do the job as well or better? At the end of our discussion, we decided to refocus and continue. We could still use the same methodology but apply it differently. Our new angle would be to have the tutors work with my English 03 class in the Blackboard site. The tutors would be more like in-class supplemental instructional assistants than like drop-in tutors. They would work with the same groups of students all semester and hopefully, develop a rapport with their students that would influence students’ to work on and improve their writing. So for the rest of the spring semester, we continued to research and evaluate technologies and best practices for online writing tutoring and we designed a training program for the tutors. During the summer 2006 semester, we trained the tutors and supervised their online interaction with students. Tomorrow, I’ll post a narrative account of the first training session.