Evaluating the quality of the impact social good organizations are claiming to create can be challenging. If you’re not an expert in their chosen cause area or region of the world, and if you’re not familiar with the needs and cultural dynamics of the clients they serve, it can be hard to know where to start. For philanthropists, funders, and the teams that support them, we’re going to go ahead and let you off the hook a little bit – frankly, it’s not your job to know!

There are so many factors to consider and such a critical eye required that it takes some deep domain-specific expertise to make a quality assessment of an organization’s impact. That expertise is truly the domain of measurement & evaluation specialists. Impact-minded funders should look to these specialists for guidance in their decision making process. That said, here are a few “green flags” to look for when evaluating the quality of impact data:

SMARTIE KPI Frameworks

All Key Performance Indicators (KPIs) that social good organizations build into their operations and report on should follow some version of the SMARTIE format. SMARTIE is an acronym that stands for Specific, Measurable, Achievable, Relevant, Time-Bound, Inclusive, and Equitable.

Specific

‍How clear are the outputs, outcomes, or impacts being created? How well-defined is the Theory of Change? Beware of confusing or nebulous language here. It’s easy to hide behind “fluffy” metrics, since most people (including, often, those working inside the organizations) are unlikely to challenge them. The less specific the KPIs, the more likely they are to be vanity metrics, and the less likely it is that the organization presenting them truly understands the issues their programs are seeking to address, and the less effective they’re likely to be in achieving their mission. If they can’t clearly explain their goals in terms a 5 year old would understand, it’s a sure sign they’re a bit unclear on it themselves.

Measurable

‍How will we know if we’re successful? We often need quantitative data to tell us what happened. Did we train enough people to successfully get jobs? How many wells were built and how many were still in operation after one year? While it can be nice to have very lofty goals, they’re often not measurable. An organization could have “changing the world” as a goal, but how will they know if they’ve affected any change if they don’t define how they’ll know if they’ve actually been successful?

Some program outcomes are easier to quantify than others, but all organizations should be working to measure what they can. While quantitative data can tell us what happened, qualitative data is better at telling us how and why something happened. Even service-oriented programs with largely qualitative results should endeavor to track quantitative impacts as well. In most cases, a healthy balance of qualitative and quantitative results is what’s required.

Achievable (and ambitious)

‍How does the scope and scale of the KPIs connect with the sophistication, expertise, and resources of the organization? Organizations should choose indicators that challenge them to do and be better. They should set their own standards of excellence, and seek to meet, if not exceed them. SMARTIE goals should always present a challenge for the organization, with “success” falling somewhere around 75-80% attainment. If goals are too ambitious as to be unachievable, the organization will never be successful by its own definition of success.

Relevant

‍How aligned are the metrics with the organization’s overall mission and ethos? How aligned is it to the best thinking in their chosen cause area? You wouldn’t necessarily want to measure a cancer research nonprofit’s ability to train children in schools, but for literacy organizations, or those supporting family stability or adolescent health issues, it might make perfect sense.

That’s a pretty clear-cut example, but there are plenty of cases where relevancy becomes less obvious. For example, nutrition programs are proven to produce better educational outcomes. It’s not enough just to build schools in third world regions. Truly successful programs like those require wraparound services like health clinics, water & sanitation, and secure infrastructure as well. It could make perfect sense for a literacy organization to fund anti-parasitic inoculation programs, since in some regions of the world, parasites are the primary reason school children stay home sick. Relevant indicators for a literacy organization could actually include the impact of inoculation programs on childhood reading.

Time-Bound

‍How long will it take to achieve the program’s goals? Under what circumstances might it require a longer or shorter timeframe than originally planned? Timing of KPI attainment must be realistic. If the program completion date doesn’t have clear time, event, or KPI-bound failure or success conditions, the odds of mission attainment may be relatively low.  

Inclusive

‍How open are the programs to people from various backgrounds, ethnicities, etc. There may be good reasons why certain groups are excluded (specific cases of trauma for example), but those are few and far between. It’s important to examine inclusivity, and make sure the policies pass the “smell test.” The metrics chosen should account for all important groups to be represented and measured.

Equitable

‍Do the chosen indicators fair for all types of program participants? How do they work to elevate low-performing clients, as well as honor their successes? Most effective programming relies on careful consideration of, and indeed, the authentic participation of the populations they’re looking to support, and do robust contingency planning to account for all possible scenarios.

Contextual Awareness

How well does the program design consider the needs, wishes, and unique attributes of the target population and how do the chosen metrics reflect this?. The strongest cases of this conduct participatory design exercises that invite the program participants into the planning process. Participatory design can and should be used with virtually all programs. When participants are brought in to help with designing the program, there’s a much better chance that the KPIs developed will be SMARTIE ones.

There are many powerful examples out there where this has been done to tremendous effect. Water and Sanitation (WASH) projects frequently cause more harm than good and reduce trust in aid organizations because they fail to consider the specific needs and cultural norms of the client population. Listen to the example below from Chris Bessenecker’s work with the Peace Corps in Nicaragua:

‍

Understanding the unique needs and cultural norms of the target population is just part of the solution. Another important consideration is the quality and longevity of the solution provided. For example, two organizations that provide free wheelchairs to people in rural communities may report equivalent metrics, but without a long-term view of the programs, you’d never know that one organization’s wheelchairs were extremely durable, while the other’s fell apart within months, resulting in program failure and unwanted plastic waste strewn about the villages where they operate, creating further problems.

So what does this all mean...

This is by no means an exhaustive list, but hopefully it’s a starting point. As impact-minded funders, you're not expected to be deep subject matter experts in every cause area and region of the world - how could you be! You can however learn to start asking the right questions, and to keep asking them until you're satisfied that you understand the reasons and justification for the approaches being taken, the results being reported, and to be aware of any indirect impacts of the work being done.

And don't worry - this is where Altruous comes in! We're here to support you in the research process, and to help make sure you're having the greatest possible impact (and minimal harm) with your philanthropic investments.

Interested in talking more about impact measurement and KPIs in the nonprofit space? Contact us, and we’ll be happy to speak with you further.