What Was Learned...

The purpose of the CCDD project was to advance understanding of the contributions of academic language skills, perspective taking, and complex reasoning to reading comprehension for students in grades 4-8, and to develop, refine, and test the efficacy of programs that target the development of these skills for a general population of students and for struggling readers. While data analyses will continue for some time, key findings are described below.

about the Theory of Deep Reading Comprehension:

The CCDD project team defined reading comprehension as the ability to extract information from written text, and construct a plausible interpretation of the author’s meaning. As students move from elementary into middle grades, the texts they are expected to comprehend change significantly.  When students are learning to read, language is intentionally kept simple and content is familiar from their everyday environments. But in the middle grades when students are expected to read to learn,  content in text is chosen precisely because it is unfamiliar. In English Language Arts and social studies classes, students are expected to see the world through the eye of characters in novels, or people who lived at another time. In mathematics students are expected to extract mathematical relationships from word problems, and in science they are expected to understand phenomena that are unseen and often counter intuitive. The project team hypothesized that this kind of comprehension requires knowledge of academic language, the ability to take perspectives other than one’s own, and the ability to reason critically.

Analysis of CCDD data confirms that each of these three factors helps to predict measures of deep reading comprehension (LaRusso et al., 2016). Academic Language, measured by the Core Academic Language Skills instrument developed in this project (Uccelli, Barr, et al., 2015), demonstrates a particularly powerful influence on reading comprehension; powerful enough to explain 12% of the variance even after grade, English-proficiency classification, SES, word reading fluency, and academic vocabulary had been included in the model (Uccelli, Galloway, Barr, Meneses, & Dobbs, 2015).

about STARI's Effectiveness:

STARI was tested in a large-scale randomized trial across four districts in the 2013-2014 school year. Eligible students were chosen by lottery (random assignment), and the RISE and GISA assessments were used to measure impact.

Evaluation of STARI was led by James Kim (Harvard University). Results published in: Kim, J. S., Hemphill, L., Troyer, M. T., Thomson, J. M., Jones, S. M., LaRusso, M. & Donovan, S. (in press). Engaging struggling adolescent readers to improve reading skills. Reading Research Quarterly.

Preliminary results:

• STARI students outperformed students in the control group on all six RISE subtests, even though the large majority of control group students received other reading interventions.
• STARI students scored 8 percentile points higher, on average, in both Efficiency of Basic Reading Comprehension and Word Recognition and 7 points higher in Morphological Awareness than control students.
• STARI students substantially reduced the gap between their reading skills and the skills of classmates who scored proficient on the state reading/ELA assessment.

ITT (“intent-to-treat”) represents the​ ​impact for all students who were assigned to STARI, regardless of the amount of curriculum they completed. But the amount of the program students actually covered mattered.​ ​TOT (“treatment-on-the-treated”) represents the​ ​impact​ ​when controlling for engagement with the program, as measured by the percent of workbook pages on which students had done any work.

​

Figure 1. Impact of STARI on the RISE assessment components and the GISA in 2013-2014. To make comparisons across subtests simpler, comparison group performances have been standardized at the 50th percentile. +p < .10; *p < .05; **p < .01

​

Figure 2. STARI Effect Sizes on the RISE and the GISA in 2013-2014. +p < .10; *p < .05; **p < .01

about STARI Implementation:

We know that the amount of the program students actually engage with makes a big difference. On most dimensions, outcomes from the program almost doubled when engagement with the program was taken into account. We also know, however, that many students in this population of struggling readers are more frequently absent from school for a variety of reasons. It is therefore critical that STARI be scheduled to maximize the chance that students will be present, and that adequate time be devoted in the schedule for the STARI program. If possible, first and last period of the day should be avoided, as should assignment to STARI as an after-school activity.

In addition, most teachers in middle grades (unless they are reading specialists) have little to no experience with teaching reading. The STARI program includes very detailed lesson plans to guide teachers step by step. But schools will likely be more successful with the program if teachers are given additional opportunities for PD and for working with colleagues who are also teaching the program.

about Word Generation's Effectiveness:

Word Generation is not only aligned with the established research base on effective instruction for supporting vocabulary, writing, and reading comprehension, but it has also demonstrated positive and significant results in several research studies, including the CCDD project. A full summary of Word Generation findings, including those from prior studies, can be found on the Word Generation website.

As part of CCDD, Word Generation was studied across four districts during the 2012-2013 and 2013-2014 school years. Twenty-five schools were randomized within pairs that were matched on multiple variables (such as size, socio-demographic characteristics, etc.), and 7,773 students in grades 4-7 participated in the study. Fourth and fifth grade treatment classrooms were provided WordGen Elementary materials. Middle grades treatment classrooms were provided a selection of units from the original Word Generation program (now called WordGen Weekly) and grade-specific Social Studies Generation and Science Generation units (Jones, et al., under revision).

Numerous analyses were conducted using CCDD’s experimental evidence. Tables 1 and 2 include some overall results from the 2013-14 school year.

​

As these results show, elementary students made significant gains in taught vocabulary, perspective articulation and positioning skills, academic language skills, and deep reading comprehension, while students in the middle grades showed significant gains in taught vocabulary perspective positioning skills, and deep comprehension. Larger impacts in the elementary grades can potentially be attributed to the greater “malleability” of younger students (Jones, et al., under revision). Another possible explanation is that implementation may have been easier in the elementary classrooms, where students generally remain with one teacher all day, than in the middle grades, where successful implementation is dependent on coordination among teachers in each of four content areas. Also, as interviews indicated, elementary teachers appreciated the new curricular resources, since they have little prescribed curriculum, whereas 6th and 7th grade teachers have prescribed curricula to cover and content standards to meet, making it more challenging to find sufficient instructional time for implementing WG (Jones, et al., under revision).

In addition, analysis of CCDD data shows specific benefits of Word Generation on deep reading comprehension, as measured by the Global Integrated Scenario-based Assessment (GISA). Academic language, perspective taking, and complex reasoning all explained small but significant amounts of variance in end-of-year GISA scores, suggesting that these domains should be given greater attention (LaRusso, Kim, Selman, Uccelli, Dawson, Jones, Donovan, & Snow, 2016).

More nuanced results arose from further analysis of the CCDD data. For example, the results reported above reflect the results obtained in the second year of the study. Impacts were stronger in year two than in year one, which makes sense given that, in the first year, teachers were introducing new practices, and student materials were still open to revisions in response to feedback (Jones, et al., under revision). Disaggregated results show that, in general, effects were larger in classrooms that implemented more of the curriculum.2 For example, students in the highest implementing elementary WG classrooms performed better in taught vocabulary (ES=.42, p<.001), perspective positioning (ES=.22, p<.01), and students in the highest implementing middle grades classrooms performed better on taught vocabulary (ES=.32, p<.001). Implementation challenges leading to variation in classroom implementation levels include things like lack of time in schedule, multiple new programs/curricula at once, and time lost to testing and test prep. These challenges are not unique to Word Generation; not only did control teachers report similar challenges, but they are common in implementation of Tier 1 programs more generally (LaRusso, Donovan, & Snow, 2016).

Additional Analyses (Preliminary)

Additional analyses have yet to be published, but preliminary results indicate that Word Generation classrooms were rated as having significantly higher quality interactions than control classrooms for the following dimensions: Regard for Adolescent Perspectives, Content Understanding, Analysis and Inquiry, Quality and Feedback, Instructional Learning Formats, and Instructional Dialogue.3 Further, Word Generation teachers had these higher quality classroom interactions even when implementing other curricula, indicating that changes in improvements in teachers’ practices carried over even when Word Generation was not being implemented (LaRusso, Jones et al., SREE, 2016). Figure 1 illustrates these findings.

Figure 1: WG Impact on Classroom Interaction Quality

​

Preliminary results from the second year of the randomized control trial also suggest that the English-learners had significantly higher gains than their non-EL peers in social perspective articulation and academic language skills, suggesting that the Word Generation program can help students catch up to their peers on these dimensions (Kim et al., SREE, 2017).

Another set of preliminary analyses explores whether Word Generation impacts vary by students’ initial skill level. Results suggest that 4th and 5th grade students with higher initial levels of reading comprehension made greater gains on the GISA when they were in WG classrooms. However, these results contradict findings from the Word Generation efficacy trial (discussed on the WordGen website), which suggested that students with lower baseline vocabulary scores showed greater growth on standardized test scores than their peers (Lawrence, Francis, Paré-Blagoev, & Snow, 2016).

about Word Generation Implementation:

WordGen Elementary: Overall, teachers found WordGen Elementary to be fairly easy to incorporate into the regular curriculum. Unlike the middle grades, teachers in these grades generally have fewer textbooks or other materials that they must cover. And they generally appreciate that the program is well designed to address many internationally benchmarked standards (including Common Core State Standards). Students find the program very engaging, which generally makes teachers’ lives more pleasant on a day to day basis: a strong selling point.

It is worth noting, however, that many of the practices in WordGen Elementary may seem unfamiliar. Teachers may believe that teaching lists of definitions is what constitutes effective vocabulary instruction, or that young students need to be told what’s right by adults rather than developing and explaining their own positions on a controversial topic. Having opportunities for teachers to challenge these assumptions with others who are more expert in research-based literacy practices (whether they are other teachers or outside PD providers) may be beneficial.

WordGen (Middle School): WordGen Weekly has been picked up by tens of thousands of teachers across the country— the most widely used SERP program. But although it was designed to be very easy to implement—taking just 15-20 minutes once a week from math, science, and social studies teachers, and twice a week for ELA teachers (some schools use other configurations)—it can be very challenging to implement as intended, especially in the absence of teacher enthusiasm since it requires significant collaboration across groups of teachers. Math and Science teachers in particular often object to participating in the program because they do not see teaching vocabulary as part of their responsibility. As with most curricula, teachers also struggle to implement the program when there are excessive constraints on instructional time due to the influx of many other new and competing programs or initiatives, and the suspension of regular instructional activities in favor of test prep activities for significant stretches of time prior to standardized testing.

WordGen Weekly can be used very effectively in schools where teachers work well together. But it can also be used to create the opportunity for teachers to begin working together when they have not done so before. Feedback suggests that teachers in some schools appreciate having something substantive as the focus for collaboration, rather than focusing on students and their behaviors, or administrative requirements. The program is very well received when teachers are interested in the opportunity it provides, but schools should expect to invest time in building teacher community if it does not currently exist. Otherwise the program is unlikely to produce the intended benefits. Students generally find the program very interesting. It gives them opportunity to have a voice on important social topics and to debate with peers. Teachers have reported that is changes the relationship between the teacher and students, and among students, in very positive ways.

SciGen and SoGen go more deeply into content knowledge, and science and social studies teachers appreciate these materials for that reason. They are not full curricula, but they target topics that most teachers of these subjects must teach at some point—topics that often pose pedagogical challenges. Most students and teachers consider them more interesting treatments of topics than alternative materials, but teachers need to be willing to do the integration into their curriculum in a way that a full, packaged curriculum does not require. They are designed for coherence with WordGen Weekly, and thus have embedded once a week activities for other content areas, but the science- and social-studies-focused materials can be used successfully independently of those additional activities, opening up the possibility that they can be adopted in schools where only the social studies and/or science teachers choose to use them.

Moving forward

SERP’s WordGen Elementary program is part of an IES contracted study of the impact of academic language programs on English learners and other struggling readers in grades 4 and 5. The single year implementation for the study is the 2017–2018 academic year. SERP is enthusiastic about pursuing other work to further test and expand the program of work developed through CCDD should funding opportunities arise. We will continue to offer summer institutes at Harvard University for both STARI and Word Generation. We are also able to provide on-site PD upon request, tapping coaches who were trained during the grant period and/or teachers who taught the program skillfully.

Data from the project continue to be analyzed, presented at conferences, and published in journals. SERP will update information on the websites and send out newsletters as new research is published. More often than newsletters, SERP shares information via Twitter and Facebook, and recommends specific units depending on current events (“The Supreme Court will rule on affirmative action in 2016. Debate this issue in class with WordGen Weekly Unit 2.02!)

We are always enthusiastic to hear about successes and challenges from those using SERP materials from around the country! Email us at the address below.

Notes

​1Vocabulary was a measure of knowledge of word meaning from a sample of target words from the Word Generation curriculum. Perspective articulation and perspective positioning were measured using the SPTAM, which asks questions based on social scenarios commonly occurring in middle school. Academic language skills were measured using the Core Academic Language Skills Instrument (CALS-I), which focuses on linguistic features common in school content discourse but not necessarily common in casual conversation. Deep reading comprehension was measured using the GISA, an ETS scenario-based assessment that asks students to evaluate, integrate, and synthesize source materials for, as an example, a fictional class discussion.

​2The level of implementation in Word Generation classrooms was calculated in two ways. Jones, et al. (under revision) did so by calculating levels of workbook exposure, defined as the total workbook activities with work done by any student in the class, and found exposure levels of 57% for fourth- and fifth-grade classes and 46% for sixth- and seventh-grade classes. LaRusso, Donovan, and Snow (2016), calculated classroom implementation levels using the average number of activities completed across all students in each class, finding average program engagement levels of 40.2% for fourth- and fifth-grade classes and 31.2% for sixth- and seventh-grade classes.

​3These dimensions were evaluated using the Classroom Assessment Scoring System, Upper Elementary, which is an observational instrument used to assess classroom interactions (CLASS; Pianta, Hamre, & Mintz, 2012).

References

Jones, S. M., LaRusso, M., Kim, J., Kim, H. Y., Selman, R., Uccelli, P., Barnes, S., Donovan, M. S., Snow, C. (under revision). Experimental effects of Word Generation on vocabulary, academic, language, perspective taking, and reading comprehension in high poverty schools.

Kim, H. Y., LaRusso, M., Jones, S. Donovan, S., & Snow, C. (2017). Reducing the Academic Inequalities for English Language Learners: Variation in Experimental Effects of Word Generation in High Poverty Middle Schools. Presented at Society for Research on Educational Effectiveness, Washington, D.C.

LaRusso, M., Donovan, S., & Snow, C. (2016). Implementation challenges for Tier One and Tier Two school-based programs for early adolescents. In B. Foorman (Ed.), Challenges to implementing effective reading intervention in schools. New Directions for Child and Adolescent Development, 154, 11–30.

LaRusso, M., Jones, S. M., Kim, H. Y., Kim, J., Barnes, S., Donovan, S., & Snow, C. (2016). Impacts of a discussion-based academic language program on classroom interactions in 4th through 7th grades. Presented at the Society for Research on Educational Effectiveness, Washington, D.C.

LaRusso, M., Kim, H. Y., Selman, R., Uccelli, P., Dawson, T., Jones, S., Donovan, M. S., & Snow, C. (2016). Contributions of academic language, perspective taking, and complex reasoning to deep reading comprehension. Journal of Research on Educational Effectiveness, 9(2), 201-222.

Lawrence, J. F., Francis, D., Paré-Blagoev, E. J., & Snow, C. E. (2016). The poor get richer: Heterogeneity in the efficacy of a school-level intervention for academic language. Journal of Research on Educational Effectiveness. http://dx.doi.org/10.1080/19345747.2016.1237596

Uccelli, P., Galloway, E. P., Barr, C. D., Meneses, A., & Dobbs, C. L. (2015). Beyond vocabulary: Exploring cross‐disciplinary academic‐language proficiency and its association with reading comprehension. Reading Research Quarterly, 50(3), 337-356.