I learned plenty from this article by two professors at the Harvard Business School. The title of their paper: What Impact? A Framework for Measuring the Scale and Scope of Social Performance. California Management Review, May 1, 2014.
Don’t be off put by the title. They make some strong points regarding whether community organizations are in the position to be measuring outcomes rather than outputs. They make a distinction between organizational mission as opposed to operational mission, and double-down on the importance of scope and scale.
Authors: A. Ebrahim and V. Kasturi Rangan are two professors at the Harvard Business School
ABSTRACT: Organizations with social missions, such as nonprofits and social enterprises, are under growing pressure to demonstrate their impacts on pressing societal problems such as global poverty. This article draws on several cases to build a performance assessment framework premised on an organization’s operational mission, scale, and scope. Not all organizations should measure their long-term impact, defined as lasting changes in the lives of people and their societies. Rather, some organizations would be better off measuring shorter-term outputs or individual outcomes. Funders such as foundations and impact investors are better positioned to measure systemic impacts.
Important point
It is not feasible, or even desirable, for all organizations to develop metrics at all levels of a results chain, from immediate outputs to long term societal impacts.
Demonstrate Results
We need to keep in mind that the word “impact” is a term used along with an emphasis on “transparency, more bang for a buck, return on investment, accountability etc.” Many organizations (start-ups nonprofits etc), end up chasing impacts dictated by external parties that don’t properly reflect the true impact of their work. In other words for one thing, it is ‘social impact’ that is key.
Conventional wisdom says you need to measure impacts as far down the logic chain as possible. But Ebrahim asks, “does this make sense for all social sector organizations?
This attention to impact, following on the heels of accountability, is mainly driven by funders who want to know whether their funds are making a difference or might be better spent elsewhere.
If social purpose organizations rely to heavily on measuring impact strictly for the reasons of proving accountability to funders, in other words showing “bang for the buck”, then Ebrahim notes this places, “too much emphasis on outcomes for which the causal links are unclear, thus reflecting more of an obsession with institutional expectations of accountability to funders than an interest in actually finding ways of improving services and results.”
The crux of Ebrahim’s argument
- Conventional wisdom in the social sector suggests that one should measure results as far down the logic chain as possible, to outcomes and societal impacts.
- This expectation is based on a normative view that organizations working on social problems, especially if they seek public support, should be able to demonstrate results in solving societal problems.
- Yet it is worth considering whether, and to what degree, such measurement makes sense for all social sector organizations.
An example: Red Cross Doctors Without Borders
They are engaged in emergency relief work. Measuring the work is straight forward: count the timeliness and delivery of emergency supplies such as tents, food water and medical supplies, count also the numbers of people reached.
Emergency relief is thus typically measured in terms of activities and outputs.
The links between inputs, activities, and outputs follow logically: the organization plans its requirements of supplies and staff (inputs), the logistics for delivering those supplies (activities) in order to provide relief to the people most affected by the emergency (outputs). When the effort is well planned and executed, the program will be able to orchestrate activities that lead to measurable outputs.
What about the outcomes?
Outcome measurement, on the other hand, requires answers to a more complex causal question: Are the activities and outputs leading to sustained improvements in the lives of affected people?
Outcome measurement is less common and more difficult to do, given that organizations have the most control over their immediate activities and outputs, whereas outcomes are often moderated by events beyond their organizational boundaries.
For example the emergency relief organization that has done excellent work during and after a natural disaster might still fall short on outcomes of rehabilitating and resettling those displaced from their homes and livelihoods, especially if those outcomes depend on extended coordination with local governments, businesses, and other NGOs.
Connecting outcomes to societal impacts, such as a sustained drop in poverty in the region, is even more complex due to the number of additional factors at play — involving the larger political, social, cultural, and economic systems—that are beyond the control of any one entity. In short, outputs don’t necessarily translate to outcomes, and outcomes don’t necessarily translate to impact.
Aravind eye hospital in India tight linkage of outputs to outcomes
An eye hospital in India performed over 340,000 surgeries, mostly cataract surgeries, its outputs providing vision correction surgery to over 3 million individuals since opening in 1979. The outcome is that vision problems will be satisfactorily cured. Indicator showing rate of complications, declined year after year.
“Even given the tight linkage in the hospital’s operations between outputs and outcomes, the organization assumes but does not measure impact—that individuals with recovered eyesight from cataract treatment will be able to lead productive lives once again and thereby contribute to society. While this assumption seems reasonable, the organization has cautiously stayed away from making that leap and seeking to take credit for impacts such as reduction in poverty, or increased well-being etc …
Under what condition is it possible to measure outcomes?
… More generally, measuring outcomes is possible under two conditions that are uncommon in the social sector: when the causal link between outputs and outcomes is well established, or when the range of the integrated interventions needed to achieve outcomes are within the control of the organization.”
Harlem Children’s Zone (HCZ)
Educational and community supports for children (Kindergarten to Grade 12)
Unlike Aravind which focuses on a singular intervention, HCZ attempts to provide a “pipeline” of interventions “from cradle to college to community” that collectively enhance the chances of a child in Harlem making it from school to the workplace and on to self-sufficiency and responsible citizenry.
HCZ’s “theory of change”
HCZ’s “theory of change” is that educational support has to be continuous from preschool through high school, and it has to be supplemented by extracurricular and community support to solidify the young person’s all-round development.
HCZ has concentrated its activities in a narrow geographical region of nearly 100 city
blocks of Harlem, under the assumption that it will be better able to control the child’s overall environment
In 2011, the sixth graders in its two main charter schools had shown significant improvements: approximately 80% were at or above grade level in statewide math exams, and between 48-67% (depending on the school) were at or above grade level in English. Moreover, 95% of seniors in public schools who attended HCZ after-school programs were accepted into college.
The grade level metrics are primarily output measures, while college acceptance may be considered an outcome measure.
The time horizon for these interventions is long (5 to 19 years), and the organization is undertaking longitudinal studies to better assess its results
Even then drawing a causal link between HCZ’s interventions and longer-term outcomes such as lifetime incomes of its graduates, and impacts such as a decline in poverty in Harlem, remains complicated due to numerous social and economic factors that HCZ cannot control.
A key distinction between Aravind and HCZ is how they make the leap from outputs to outcomes. Aravind can measure outcomes because of the tight causal linkage between its outputs (corrective surgery) and outcomes (quality vision), which hinges on the quality of its surgery. HCZ can measure outcomes because it is able to vertically integrate a comprehensive set of interventions (a pipeline from cradle to college) that it controls in a tightly bounded geographic space (100 city blocks) (p. 125).
Clearly, all organizations should be capable of measuring the outputs of their operations. However, only some will be able to go further to make credible and measurable claims about outcomes. This is possible under two conditions:
- the organization implements a narrow scope of activities where the causal link between outputs and outcomes is clearly established through evidence (e.g., Aravind’s eye surgeries lead to improved vision); or
- the organization implements a broad scope of activities that is vertically integrated to increase control over outcomes (e.g., HCZ’s cradle to college pipeline) (p. 128).
