Evaluation of EVERYBODY WINS! DC Power Lunch Program
Executive Summary by Meredith Smith
The growing recognition that too many children read poorly or scarcely at all has prompted an increase in programs in which adult volunteers seek to instill a love of books and learning in disadvantaged students. There has been little research, however, on the effectiveness of such efforts.
The U.S. Department of Education’s Office of Planning and Evaluation thus sponsored a study of Everybody Wins! DC, a privately funded reading and mentoring program in which various professionals—from the Congress, the businesses community and nonprofit groups—read once a week with disadvantaged elementary school children at lunchtime. Everybody Wins! works with teachers to select as many students as possible who are reading below grade level, who would benefit from the attention of a caring adult and who are not read to at home (1). In the 1998-99 school year, when the evaluation was done, this Power Lunch Program served 1,200 students in 10 Washington, D.C.-area schools that receive aid under Title I of the Elementary and Secondary Education Act. The rigorous study, employing a large, randomly selected student sample and conducted with design and evaluation guidance from the Department of Education and the American Institutes for Research,(2) found that the program imparted notable benefits to the students and especially to its primary targets, the poorest readers. For example, teachers reported that Everybody Wins! students who read below grade level showed increased enthusiasm for books and improved both their academic performance and their classroom behavior. (3)
Specifically, the children’s teachers found that by the end of the school year:
For all students in the Everybody Wins! program (4) , teachers found that:
The Everybody Wins! program is not designed to drill students in phonics or whole word reading methods or to otherwise systematically and directly teach reading techniques. Those are teacher responsibilities. The study, therefore, did not attempt to test students on a standardized reading exam. Rather, Everybody Wins! is intended to impart a love of books and learning, to instill positive attitudes toward reading, which research has long indicated to have important effects on student literacy. This initial study, especially with its academic gains for the poorest readers, indicates that the program is making important strides on these fronts, that participants are reading more and thus learning more.
This is not to suggest that all aspects of the study showed dramatic and positive results. Teachers, for example, found no significant difference between the overall treatment group and the overall control group on how motivated students are to read. However, the program’s chief targets did show gains in this area. Poor readers in the Everybody Wins! Power Lunch program, teachers reported, were more than twice as likely to be “very motivated to read” (14%) as low readers in the control group (6%). In the same vein, while teacher reported gains in Power Lunch students’ attention spans, they found little difference between treatment and control groups in other reading-related skills, such as listening comprehension, vocabulary and ability to articulate thoughts. It should be noted, however, that in separate focus groups, volunteer mentors reported that their students made their most significant gains—taking on more difficult material, developing more confidence, choosing longer books—in the program’s second school year, which was beyond the scope of this study. A few students, focus group mentors reported, even went from nonreaders to readers in the second year. It should also be noted that only students who participated fully in the program (meeting with mentors more than 15 times over six to seven months) were included in the treatment group for the purpose of this analysis. Students who met with mentors less frequently (29 students) tended to score lower than the control group in teacher reports, suggesting that disappointed by a mentor may hurt student reading attitudes, at least in the short term.
After one school year, the study found that the program’s principal targets—students reading at the lowest levels—were the largest beneficiaries of the Power Lunch program. Both teacher reports and student self-reports found significant program gains for students, particularly in reading attitudes, self-esteem, library use and academic performance. Results also indicated that students who have more frequent meetings with their mentors benefit the most from program participation and that infrequent or inconsistent meetings may have a negative effect on student attitudes toward reading.
The study’s strengths included a scientifically rigorous design, very low levels of study dropout (6%) and high survey return rates (94%). The primary study weakness resulted from the need to randomly assign students to treatment or control groups from within the same classroom. Students who were not in the program could have been influenced by their peers in the program, creating some risk that the study actually underestimated the program’s benefits. Overall, results are very promising and indicate that students, especially those most in need of assistance, are benefiting from their participation. Further research is needed, however, to confirm these findings, to examine whether benefits increase over time and are sustained, and to better understand the general relationship between reading attitudes and literacy. Following are key data from the study presented in bar charts.
(1) The program cannot exclude disadvantaged students who already read at or above grade level but who are eager to participate; they constitute a minority of participants.
(2) The evaluation was conducted with a total of 223 pupils, or nearly 20% of the student universe,spread among seven schools. To assure reliable results, coin tosses determined which students were matched with a mentor and which went in a control group. Twenty-two teachers and 160 reading mentors also participated in the evaluation. The study used three survey instruments administered at the beginning and end of the school year. The teacher survey was developed by the program evaluation team and pilot-tested with teachers. Students reported on their own reading attitudes by responding to the Elementary Reading Attitudes Survey (ERAS) (McKenna, Kear, and Ellsworth 1995) as well as to a self-esteem scale, modified for younger readers, from an evaluation tool used by the Big Brothers Big Sisters Program (P/PV, 1998).
(3) See bar chart 1 below.
(4) See bar chart 2 below.
(5) See bar chart 3 below.