Impact measurement – its confusing…
By Sandy Blackburn-Wright, Managing Director, Social Outcomes
We are told at every turn that we need to measure impact, whether we are a not-for-profit (NFP), philanthropic foundation, business, or government department, and it’s all very confusing. There are lots of measurement tools and approaches, and we’re not always sure whether we should use these, or build something ourselves, so we end up doing a bit of both. Here are a few thoughts on what I’ve learnt as I’ve waded through all the options.
The place to start is who needs to know what, when, and why. While there are many different approaches to measuring impact, from the complex randomised control group of some Social Impact Bonds, through to GIIRS (Global Impact Investment Reporting System) and SROI (Social Return On Investment) – its actually horses for courses. We all understand the importance of moving towards measuring outcomes, not just outputs – but in reality, we may need to focus on both.
Inputs are the resources we bring to addressing an issue, activities are the work we do, and outputs are a count of products or services provided. All of these measure the work of our organisation. So if we are running a training and support program for at-risk women wanting to re-enter the workforce, we have trainers, a venue, training materials, a catering budget, etc, and these are our inputs. The activities we undertake include curriculum and materials development, identifying women to attend the training, and events management. The outputs we report are the number of women trained, the number of courses run, and perhaps some feedback on how the women found the training. The outcomes of our training work is expressed in the changes in the lives of the women we work with, not the workings of our own organisation. So when we focus on our beneficiaries, we want to know – are they finding and sustaining work, are they going back to school, are they engaging in community, are they finding a stable place to stay – are their lives improving as a result of our work with them? Our theory of change is reflected in what we measure. We believe that if we give women the skills and confidence to get back into the workforce, and provide support over time once they get there, then they will be able to afford stable housing, are they thinking about the future and so on.
But in terms of inputs, activities, outputs and outcomes, who needs to know what, when, and why?
If we are the CEO or senior executive of an NFP who is delivering the programs, then inputs, activities and outputs are of extreme interest to us. We need to know if what we are doing is efficient and effective, and once we get some feedback from our key measures, we can make adjustments to our programs to improve what we do. As the CEO, we are also interested in outcomes. We need to know if we are creating a positive impact on the lives of the people we exist to serve. And if not, we need to rethink our theory of change or re-target our programs and partners.
Under current contracting and grant arrangements with government and philanthropists, NFP’s provide regular acquittal reports on inputs, activities and outputs, but the question is – do they need to know? And if they are provided with all this data, what will they do with it? Probably very little as this is the work of the CEO, executive team, and in some instances, the Board of the NFP. What should be of more interest to funders is – what impact is the work creating, and how do we know? This is where some funders are using the SROI tool. But for me, SROI is a useful organisation review tool, rather than a measurement tool that can compare the impact of different organisations. SROI practitioners will agree that you can’t compare assessment. However, it can be a useful tool to communicate impact to the general public. So as with all things, it depends on the audience.
When it comes to impact measurement, I prefer to look for relevant publicly available data, partly because it’s transparent, and partly because it’s free, with someone else doing the data collection work for you. There is huge amounts of data available in the public domain that is collected and shared by government and other agencies. For example, if you are working in education, you can access data on truancy, educational attainment, NAPLAN results, enrolments in TAFEs and universities, etc. While these aren’t perfect measures, they do show clear trends, and if you break them down to a specific geography, you can compare the communities in which you work, to those you don’t, by specific issues. So if you are a medium or large organisation, publically available data is invaluable.
If you are a small organisation and you work with a small group of people, then tracking the changes in their lives is the best approach. For example, you could say that “our social enterprise has supported 18 people to gain permanent employment over the last 24 months, none of whom have any contact with the justice system in that time, and 4 people have been able to move out of social housing and into the private rental market”. Its also worth combining these generic numbers with a narrative of the impact on one individual, with their permission of course.
This brings us to the issue of attribution – how do you know it was you? To be honest, I don’t think you ever can. You can never know whether outcomes were perfectly attributable to your work, and your work alone. It’s never that neat and clean. What we can look to demonstrate is whether we have contributed to a change. This is the kind of reporting we should be creating for our funders, but it’s also the kind of reporting funders should be creating for providers. For example, if a government agency is using a contracting for outcomes approach, then the role they play is to measure and report on those outcomes and share that back with the provider so they can enhance their activities over time. In this way, funders move from collecting information which won’t affect the outcomes achieved, to sharing information that can. Funders also move from being enforcers of contracts, to partners in the creation of those outcomes.
If you are a large social enterprise, NFP or a business that is seeking impact investment, then the evolving standards in GIIRS become important. Investors need to get a clear picture over time as to how the enterprise is performing on a financial, social and environmental basis, to assess the risks involved. These measures require substantial effort to report on, but it is appropriate where external investment is being sought. Standardised frameworks will also add real value as investors seek to compare opportunities.
Social impact bonds have really raised the bar on measurement, but it may also be that their high quality measurement approaches make them so complicated to implement that they won’t be used as often as they could. Through SIBs, we are seeing complex counterfactuals, such as randomised control groups, that provide strong evidence to both investors and government that outcomes are being achieved and can be attributed to the provider. Whilst this is key, I think we need to be asking ourselves, “what is the simplest form of measurement that investors and government can accept?”
So the rule of thumb in this growing world of measurement and transparency is to match the measure to the audience, and understand what they will do with the information you provide. Also, increased calls for measurement shouldn’t tempt us to overstate what we are achieving. Rather, lets be honest, use the simplest measures we can and seek to partner with funders in the creation of positive impact.