Not all impact is created equal, and if you’re not well-versed in nonprofit measurement & evaluation (M&E) practices, it can be hard to tell the difference. Without a strong background, or internal resources to invest in quality M&E, many organizations will present statistics in their marketing materials and annual reports that have little-to-no connection with the impact their programs are actually having.

Does this sound familiar?

“2022 was a banner year at our organization. With 95% of our annual budget going directly to programs, we’ve never operated more efficiently! For the first time ever, our programs reached more than 750 children, and all but 4 of them reported complete satisfaction with our programs. Not only that, more than 10,000 people viewed the movie we made about the cause, and we grew our social media following by 150%.”

Sounds pretty good, right? Not so fast. All of these statistics are known as “vanity metrics,” and as nice as they sound, they give us little indication of how effective an organization actually is. Let’s probe a bit deeper…

Vanity Metric 1: Overhead

Overhead can be one of the most misleading statistics any organization will report. Efficiency can certainly be a virtue, but so too is hiring key positions, and paying competitive salaries and benefits. So too purchasing and maintaining crucial equipment.

What about investment in innovation? With overhead so low, how are we actually implementing programs? Are volunteers staffing roles that should really be done by full-time employees? Even if we did have answers to all of these questions, it still doesn’t tell us anything about the quality of the work being done or how durable the outcomes are being created.

When considering overhead, it’s important to ask questions like these. As funders, we should be looking at how effective the programs are at achieving their missions, and how deep and lasting the results are.

All things being equal, reporting only 5% overhead is probably more of a red flag than a virtue. As long as they’re a solvent and following strong business practices, they probably should be spending a bit more on infrastructure.

Vanity Metric 2: Reach

It’s impressive that so many people watched the movie, and so many kids were reached, isn’t it? Of course, but things like “reach,” “attendance,” “graduates,” and “viewers” don’t tell us anything about what really happened.

It’s nice that so many children were reached, but reached how?

Were they given flyers or pamphlets, or did they participate in services over an extended period of time? Did they complete course work of any kind, and if so, what difference did those services make in their lives? Are they safer, smarter, or more confident? Did they learn any new skills, or are they reading at a higher grade level than they would have otherwise?

For people who viewed the movie, what followup actions were they asked to complete, and how many of them did so? As a result of their viewing, did they learn anything new? Did they build a stronger bond with the organization, or was the movie itself a disincentive for continued engagement with the program?

Vanity Metric 3: Client Satisfaction

We always want people to be satisfied with our services, don’t we? Of course we want positive interactions, and it’s always important to get feedback from stakeholders to drive innovation. However, satisfaction in the social sector can be misleading.

Sometimes organizations need to do things to serve clients as part of their drive to create positive social impact that those clients might not like. What about food pantries that serve only healthy options in an effort to combat obesity? Their clients might prefer less healthy options and therefore be less satisfied with the service.

Counselors may be ethically or legally bound to tell clients things they don’t want to hear. What about residential treatment or rehab programs? What if clients love the residence so much that they never want to leave? There are many programs out there that create a cycle of codependency that inhibits the very results they seek to achieve.

So, as philanthropic funders, what should we be looking for…

Impact is what happens after. Impact is what people do with the information or services they’ve received. It’s the extent to which their lives were improved, and for how long.

It’s important to examine how an organization is performing relative to its size, revenue, and stage of growth - both against its own potential, and against how other, similar programs perform. To the extent possible, it’s important to consider causation - would the outcomes an organization reports have happened anyway, without their intervention? Did their programs accelerate, inhibit, strengthen, or weaken the results their clients would’ve seen without their support?

Quality impact evaluation is hard. It’s complex, and it requires deep domain expertise and cause (and region) specific understanding. It’s because it’s hard that so many organizations understandably revert back to vanity metrics, and why so many funders accept them at face value.

For philanthropists, foundations, companies and other organizations looking to have real and lasting impact, it’s important to start by asking questions, and to keep asking them until the true picture of organizational effectiveness begins to emerge. For more on how to do this well, check out our blog on key metrics to look for when evaluating programmatic impact.