Subject: “A New Measure for Classroom Quality”

Interesting article…how do we respond to this most effectively?

Jim Connell, IRRE President

Re: “A New Measure for Classroom Quality”

All,

I thought it was telling that these discussions about “evaluating” teachers/teaching so often end at diagnosing teachers/teaching as ‘good’ or ‘bad’ as if that is the real aim of the evaluation. It seems to me (and ‘us’ — I think) that a response should try to deepen this discussion…i.e…that evaluation should accomplish (at least) two goals: 1) an evaluation of classroom instructional quality; and 2) providing the kind of insight/information about teaching needed to design effective PD to support teachers through the difficult process of changing and improving their practice.  I for one, would like to hear Julie and Anissa’s take on this.

Todd Lacher, IRRE Research Manager

Re: “A New Measure for Classroom Quality”

All:

Todd,  You hit on the exact two points that stuck for me although a bit of expansion on number two as well as a couple of other ideas.

Most important point Todd brought up and is my greatest push back on the Instructional Rounds approach; our tool is not about diagnosing and then “fixing or healing” a “sick” teacher it is about growth and support for all teachers.  It is a tool when used well provides leaders with the data that are being demanded by the public but at the same time provides a basis for all additional instructional support to grow.

1)      As you have said the measure should identify the quality of instruction or state of teaching and learning in the building, across the district, with in departments, by course level, by grade level and individually – making visible the trends that are taking place.

2)      Identifying the PD supports needed: topics, types (small group, one on one coaching, modeling, peer visits, etc) and how much

3)      Tracking and monitoring the implementation of the new learning and effectiveness of the supports provided

4)      Increasing the instructional leaders’ repertoire of instructional strategies by learning from those they are observing and having the opportunity to give credit and share those new learnings across the school.

5)      Provide an impetus for teachers to engage in reflective dialogue and work with their coaches and peers to improve their practice.

My thoughts.

Julie Broom, IRRE Director of Instruction

Re: “A New Measure for Classroom Quality”

It struck me as I was reading that the point seemed to be only to determine who was a “good” teacher and who was “bad” without any thought as to support for growth.  Too often, articles focus on teachers teaching rather than students learning.  I had a conversation with a literacy coach this past week about how the title “teacher” no longer reflects our true goal which is to help facilitate learning of students.

I think another piece is that there seems to be a lack of definition and agreement often in what “good teaching” looks like.  That’s why it always made sense to me to look more at the learning behaviors of students.  While the instructional strategies can vary and look differently from one class or one teacher to another, learning behaviors are more universal.  They are also the true end result.  It also seems that teachers react more positively and can have much more effective and lasting conversations that change their own practice when the focus is on impacting student learning and what that should look like rather than saying their own behaviors should like a specific way.

Those are my thoughts.

Anissa Collins, IRRE Associate Director of Instruction

We here at The Forum would love to hear what YOU think.  Come join the conversation.

A recent article featured in the Chronicle of Philanthropy makes a case against the latest push for a more ‘data-driven’ philanthropy. The article, titled Measurement is a Futile Way to Approach Grant Making, traces the history of philanthropic and government efforts to use measures of investment outcomes to guide their funding decisions.  The author, William Schambra, makes some important and provocative points.  He argues that the considerable effort it takes to collect, analyze and report data from these measures — especially on the part of practitioners — simply isn’t worth it – a) because findings from these studies are rarely used by their funders; and b) because the measures used in these studies tend to be idiosyncratic and transient and therefore can’t contribute useful information to the broader field.  We agree on these points and, like the author, are disturbed by them.

Where we differ from Mr. Schambra is in how to remedy this situation.   He calls for abandoning or radically curtailing efforts to measure the outcomes of education investments.  We say change the way it’s done: measure things that really matter to practitioners and investors; report results in more timely and consistent ways; and build capacity both of practitioners and investors to use these data more effectively.

Most of IRRE’s work has been with struggling high schools trying to transform their practices to produce better results for their students.  Our partner investors and practitioners in this work care about results – but what results?  Two come to mind: students making timely progress toward successful graduation and learning what they need to move on successfully to post-secondary training or college.  So let’s measure these things and let’s make sure the data we’re collecting can be used to help students, educators and investors make decisions before the students we’re measuring and the money we’re investing are gone.  Let’s also make sure we all measure these outcomes and their leading indicators similarly enough across initiatives so we can learn something from each others’ results.

Some Examples

IRRE and our partners use several leading indicators of graduation rates and post-secondary success:  1) progress toward graduation (the number of required credits for graduation divided by the number of credits the student should have accumulated by that point in their high school career); and, for college and career readiness, successful completion of gateway courses to post-secondary opportunities (i.e., passing or better grades in courses required for entry into 2-year and 4-year colleges and/or into defined post-secondary employment and training opportunities).  These indicators are neither exhaustive nor optimally precise; but, they illustrate how we can measure something meaningful to practitioners and investors (and students themselves) on a regular basis; they are available from existing student records (low burden on practitioners); and they can be calculated similarly enough across education initiatives to have results be compared for knowledge building purposes.

We have seen the utility of these kinds of measures enhanced by adding indicators of key practices known to affect these outcomes.  In particular, measures of classroom teaching and learning that provide a common definition and focus for instructional improvement.  Admittedly the burden on practitioners – instructional leaders and classroom teachers – increases by adding these measures to the initiative’s outcomes “dashboard.”  But the payoff can be significant — both to practitioners and to investors and ultimately to the field if these implementation measures are used across multiple initiatives.

IRRE and its school and district partners use a classroom visit protocol (loaded on PDAs, iPads and smart phone devices) that measures levels of engagement, alignment and rigor (EAR) in the teaching and learning observed during 20-minute classroom visits. With this system, school and district leaders gain immediate access to summary and individual classroom reports on the results; and are using these data to enrich and focus data-driven dialogue around student learning outcomes to design professional development and evaluate its effectiveness.   Data collected on this protocol in our rural, urban and ex-urban partner schools and districts are generating findings on how teaching and learning can be improved in these diverse contexts.

With actionable data on student outcomes and effective practices available the next step is to use it and use it effectively. In IRRE’s partnerships with schools and districts across the country we have learned how important it is to provide intensive supports for effective use of data at all levels of educational practice – teacher, administrator and district staff.  Building this capacity at all levels of the system has dramatically increased the perceived value of collecting the data in the first place.  By extension, similar kinds of professional development supports should and can be made available to investors as well.

In response to Mr. Schambra’s legitimate concerns with the state of outcomes-driven philanthropy, we argue for the positive benefits (relative to the burden on practitioners) of collecting certain kinds of data, reporting them in a timely fashion and supporting practitioners and investors to use them to guide educational practice and funding.  These data include: a common and limited set of long-term student outcomes of clear import to students, education practitioners and policy makers; leading indicators of these outcomes available from most student information systems; and assessments of a few critical practices contributing to those outcomes.  All three sets of measures can use common metrics so results on these outcomes can be used in comparative studies across different initiatives and time points.

Resources

Mr. Schambra correctly warns readers about “experts” who too often claim, “… all those earlier efforts failed, because they didn’t do it my way!”   The strategies we propose here are meant to be illustrative of the more general  point that taking small numbers of meaningful, and common measures of progress in a timely fashion with support provided to users of their results actually can be done and, we argue,  should be done for the benefit of practitioners and investors alike.  Below we provide a list of other sources describing different strategies to get to this same goal.

“Getting Ideas Into Action: Building Networked Improvement Communities in Education,” a Carnegie Foundation supported report. Available online at: http://www.carnegiefoundation.org/sites/default/files/bryk-gomez_building-nics-education.pdf

The Annenberg Institute for School Reform’s “Leading Indicator Series,” available at:  http://www.annenberginstitute.org/Products/LeadingIndicatorsSeries.php

The Consortium on Chicago School Research’s work identifying “On-Track” indicators for youth: http://ccsr.uchicago.edu/publications/07%20What%20Matters%20Final.pdf and their publication titled “A New Model for the Role of Research in Supporting Urban School Reform” http://ccsr.uchicago.edu/publications/CCSR%20Model%20Report-final.pdf

The FSG report “Breakthroughs in Shared Measurement and Social Impact” available at: http://www.fsg.org/tabid/191/ArticleId/87/Default.aspx?srpush=true

Researchers Without Borders’ work on measuring implementation fidelity and program enactment: http://www.researcherswithoutborders.org/projects/measuring-enactment

Three Basic Psychological Needs

The intent of IRRE’s work is to create conditions in schools and school systems (for students and adults) that most powerfully support their meeting three fundamental psychological needs:

• competence,
• autonomy and
• relatedness.

These three needs are the cornerstone of Self‐determination Theory (SDT) – as developed by Edward Deci and Richard Ryan – and mentioned in the On Performance discussion (see also http://www.psych.rochester.edu/SDT/).

For students, our core strategies are designed to create conditions that increase their sense that they know what it takes to do well in school and can pull it off (competence), that give them an important role in shaping their own learning – learning aligned with things they see as important and worthwhile (autonomy) and that let them know they are connected to other people in school (peers and educators) and at home (family) in ways that support their learning and development (relatedness).

According to SDT, with these needs being met, students should engage emotionally, behaviorally and cognitively in the work of school and, by doing so, do better, like school more and stay with it even when challenged. When these needs are not met, students will become disaffected – avoid cognitive challenges, withdraw emotionally or act out and do as little as possible to get by. With disaffection comes decreases in attendance, drop‐out, and poorer academic performance; with engagement, students are more likely to show up, graduate and perform better academically. These hypotheses drawn from SDT have been supported repeatedly in empirical studies of diverse student populations at elementary, middle and high school levels using measures specific to SDT and instruments and methodologies drawn from other theoretical perspectives as well.

What Does it Look Like in Schools?

Returning now to the “conditions” in schools and school systems that can promote or undermine students’ motivation and learning, SDT hypothesizes that three aspects of students’ experience in school are critical. Do schools provide their students:

• High, clear and fair expectations, and effective supports to meet them (structure);
• Opportunities to make meaningful choices in school and to make meaning of their experiences in school (autonomy support); and
• People (adults and peers) who know, respect, trust and care about them

Again, a significant body of literature exists that confirms these and closely related dimensions of schooling make a difference in students sense of competence, autonomy and relatedness; their engagement; and their doing what it takes to finish school and perform well academically

Over the past 15 years, IRRE’s reform framework First Things First (FTF) has evolved from these basic tenets. This work has articulated the changes needed in school structures and functions to strengthen these three conditions in support of student motivation, engagement and learning. The theory and research that initially inspired our work focused on students – their experiences in school, their motivation, and their engagement and learning. However, our current work includes processes that support individual-, school‐ and system level change in adult behavior. This extension of FTF has been shaped by:

• What earlier reform frameworks and their results showed were practices tied to improving the structure, autonomy support and involvement experienced by students in school;
• What motivational theory and research (and other related literatures) had to say about engaging adults to change their own and their systems’ behavior and beliefs about schools and schooling; and
• What IRRE and our partner schools and districts did that seemed to work or not work to produce desired change in students’ and adults’ motivation, engagement and learning.

An Invitation to Share Your Roots

These are our roots. Motivational theory and research provided the spark that led to bringing educators and researchers together to develop the strategies that became First Things First. We invite practitioners and reform support organization members to share with us – briefly or at length – where you came from in your work to improve schools.

Come join the conversation.

Excerpts from:

Connell, J.P., Klem, A.M., Lacher, T., Leiderman, S., & Moore, W. (2009). First Things First: Theory, Research and Practice. Toms River, NJ: Institute for Research and Reform in Education. Available online: http://www.irre.org/publications

Federal policy makers, and increasingly the American public through vehicles such as “Waiting for Superman” recognize the urgent need to make rapid and meaningful change in the educational experiences of millions of American public school students now attending low performing schools.

Unfortunately, commonly advocated responses to this need, both old and new, have shown little evidence thus far of being able to address it at scale. The federal government is currently investing billions of dollars in a variety of strategies to improve these schools through School Improvement Grants, the Race to the Top program and the Investing In Innovation (i3) Fund. These huge investments acknowledge both the need to find sustainable and scalable school improvement solutions and the nascent state of our efforts to do so.

Historically the field has looked to three approaches to meet this need. Targeted approaches focus on slices of the school improvement problem or on small percentages of struggling schools’ students. Even the best of these programs, have not, by themselves, turned many struggling schools into thriving ones. More ambitious comprehensive approaches (locally or externally initiated) have recorded some successes; but failures to implement them consistently with fidelity or sustain them across leadership transitions weaken their case for scalability. The strong wind now filling the sails of new turnaround approaches such as charter schools, replacing principals and staff, and school reconstitution comes in significant part from investor enthusiasm and local success stories, rather than consistent or widespread results. Emerging evidence reveals these turnaround approaches’ are vulnerable to the same implementation and sustainability challenges as comprehensive reform approaches.

The Active Ingredients of School Improvement

IRRE’s 15‐year history spans this entire evolution of reform thinking. We continue to believe there are ways to meaningfully improve large numbers of chronically underperforming schools; and that the active ingredients of these solutions are known. In IRRE’s view, they are:

• a rigorous, complete and continuous diagnosis of what educational practices are missing, insufficiently implemented or doing harm;

• sufficient initial and ongoing capacity to provide educators high quality training and supports for implementation of the educational practices needed; and

• accountability and management conditions for all involved that bring these two ingredients into large numbers of struggling schools with sufficient focus, force and persistence to generate and sustain meaningful change.

Each of these ingredients shows up in one or more of the prevailing reform approaches. For example, great training and supports for teachers and building leaders characterize the best of all three approaches. Several comprehensive reform models identify and seek to implement the full array of supports needed by struggling schools and their students to be successful. Flexible staff recruitment and strong accountability characterize the best of the new turnaround approaches. What remains elusive is the recipe for combining these ingredients in ways that produce consistent and sustainable results for large numbers of struggling schools.

Combining Ingredients

If there is a modus operandi for this blog it will be to provoke honest conversations about how schools, districts, foundations, governments, and organizations like ours can work together and be smarter about how these key ingredients get implemented and combined.  Too often the work of school improvement remains compartmentalized because of competition, ideology, and politics.  We believe common ground exists that can be leveraged to move this critical work forward and improve our young peoples’ educational experiences and outcomes.

Come join the conversation.