Key Questions About Philanthropy, Part 3: Making and Evaluating Grants

Throughout the post, “we” refers to GiveWell and Good Ventures, who work as partners on the Open Philanthropy Project.

This post is third in a series on fundamental (and under-discussed) questions about philanthropy that we’ve grappled with in starting a grantmaking organization (see previous link for the series intro, and this link for the second installment). This post covers the following questions:

• When making a grant, should we focus most on evaluating the strategy/intervention, the leadership, or something else? We think both are very important; for a smaller grant we hope to be excited about one or the other, and for a larger grant we hope to thoroughly assess both. A couple of disanalogies between philanthropy and for-profit investing point to a relatively larger role for evaluating strategies/interventions, relative to people. More
• For a given budget, is it best to make fewer and larger grants or more numerous and smaller grants? We currently lean toward the former. Most of the grants we've made so far are either (a) a major grant that we've put major time into or (b) a smaller grant that we've put less time into, in the hopes of seeding a project that could raise more money down the line. More
• What sort of paperwork should accompany a grant? Funders often require grantees to complete lengthy writeups about their plans, strengths, weaknesses, and alignment with funder goals. So far, we've taken a different approach: we create a writeup ourselves and work informally with the grantee to get the information we need. We do have a standard grant agreement that covers topics such as transparency (setting out our intention to write publicly about the grant) and, when appropriate, research practices (e.g. preregistration and data sharing). More
• What should the relationship be between different funders? How strongly should we seek collaboration, versus seeking to fund what others won't? It seems to us that many major funders greatly value collaboration, and often pursue multi-funder partnerships. We don't fully understand the reasons for this and would like to understand them better. Our instincts tend to run the other way. All else equal, we prefer to fund things that are relatively neglected by other funders. We see a lot of value in informal contact with other funders – in checking in, discussing potential grants, and pitching giving opportunities – but a more formal collaboration with another staffed funder would likely introduce a significant amount of time cost and coordination challenges, and we haven't yet come across a situation in which that seemed like the best approach. More
• How should we evaluate the results of our grants? Of all the questions in this series, this is the one we've seen the most written about. Our approach is very much case-by-case: for some grants, we find it appropriate to do metrics-driven evaluation with quantifiable targets, while for others we tend to have a long time horizon and high tolerance for uncertainty along the way. More

When making a grant, should we focus most on evaluating the strategy/intervention, the leadership, or something else?

We've received a fair amount of advice to focus more on supporting the best people than on supporting the best ideas, since execution is extremely important and ideas tend to evolve greatly as a project progresses. On the flip side, we believe it is often very difficult to evaluate people with any reliability, especially when working with people who don't have significant track records (as may be necessary when funding novel projects and nascent fields).

I note that I see a couple of disanalogies between philanthropy and for-profit investing that point to a relatively larger role for evaluating strategies/interventions, relative to people:

• In the nonprofit world, it is quite possible – and probably common – for a project to have enormous success according to one person and one set of values, while having little success according to another. (By contrast, in the for-profit world, there is widespread agreement on the ultimate goal of profitability.) In my view, funding an outstanding leader in a low-impact cause can easily result in a project that succeeds on its own terms but still doesn't represent an outstanding grant. In addition – as discussed in a previous post – it is often a funder's responsibility to try to attract people into particular causes and missions, rather than supporting whatever plans the most capable people have already formed.
• I also believe – weakly, and based on informal observations over the years – that it is often possible for a nonprofit to start with weak leadership and improve over time, in a way that would be much less likely in the for-profit sector. In the for-profit sector, if someone likes an organization's goal but thinks its execution is poor, they're likely to start a competitor; in the nonprofit world, they may be more likely to try to join (or otherwise help) the organization.

In general, for relatively small "seed" grants, we've had the attitude that we want to be excited about the idea or the people, but not necessarily both, since it can be very difficult to evaluate either with confidence. For the biggest bets, we believe it is important to do what we can to evaluate both.

For a given budget, is it best to make fewer and larger grants or more numerous and smaller grants?

We know relatively little about other funders’ views on this question, other than that the Sandler Foundation seems to favor fewer and larger grants compared to most foundations we've seen.

Our current thinking is based on a couple of conceptual points:

• At the moment, our scarcest resource – and the one we have to be most careful about budgeting – is time rather than money. (We believe this will often, though not always, be the case for a philanthropist.) For a given level of due diligence, making fewer and bigger grants means being able to give away more; for a given budget, making fewer and bigger grants means being able to put more attention into each grant. All else equal, we see this as one reason to lean toward making fewer and bigger grants.
• When there is very little track record for a project or team, something in the range of $100,000-$250,000 often seems like enough to (a) encourage the grantee to commit more seriously to fundraising and planning for a project; (b) improve the grantee's odds of securing funds from others; (c) give the grantee enough resources to build up a basic track record – a year or two of full-time work from a small team – which can become the basis for a later, larger grant (either from us or from another funder). We'd hesitate to make a larger grant unless we had a reasonable sense for the track record of the organization or team; we'd fear that a larger grant could be inefficient (by missing out on the opportunities for leverage just listed) and could also encourage a grantee to build capacity prematurely.
• Reconciling the above two points, the grants we've made so far have generally either been (a) major grants that we've put major time into or (b) smaller grants that we've put less time into, in the hopes of seeding a project that could raise more money (from us or others) down the line.
• At the "seed" stage, we're comfortable accounting for a very large part of a team's or project's budget, since our aim is to fund exploratory/preliminary work that may or may not lead to more. For larger grants, we keep in mind that providing too much of an organization's funding can make it dependent on us; such a situation may be necessary in some cases, but all else equal, organizations with more diverse funding bases are in better positions.

What sort of paperwork should accompany a grant?

Based on GiveWell's experience as a grantee, as well as grant proposals we've seen from nonprofits we've worked with, it appears that funders often require grantees to complete lengthy writeups about their plans, strengths, weaknesses, and alignment with funder goals. Such writeups also often include setting out specific goals that later progress can be measured against.

So far, we've tried to take a different approach: we've tried to write up the case for each grant ourselves, and we've tried to get information from grantees in the form of conversations, other informal communications, and already-existing documents rather than requesting that new materials be created. We believe this approach allows us to maintain quality control over our writeups and avoid excessively burdening grantees. Ultimately, the purpose of our writeups is to assess grants according to our own values and strategies, so it seems logical for us to develop the capacity for creating such writeups, rather than asking our grantees to do so. When it comes to setting out specific goals, we generally include a section on this, but often stick to long-term and/or highly general goals, because we think this is appropriate for many sorts of grants (more below).

We generally ask grantees to sign a grant agreement covering topics such as confidentiality, our expectations around checking in and sharing public writeups, our preferences that any original research share its data and create a pre-analysis plan, and legal issues.

What should the relationship be between different funders? How strongly should we seek collaboration, versus seeking to fund what others won't?

It seems to us that many major funders greatly value collaboration, and often pursue multi-funder partnerships. We don't fully understand the reasons for this, and would like to understand them better. Our instincts tend to run the other way: we seek to fund the best work that we believe would have trouble getting funded otherwise. All else equal, seeing that other funders are interested in an approach makes us less interested in that approach.

Sometimes, when a project fails to get interest from other funders, this can be a warning sign. Talking to funders who could have supported a project, and didn't, can help us learn about possible shortcomings and reservations we would not have thought of otherwise. On the other hand, sometimes a lack of interest from other funders simply means that we've found a neglected opportunity, which is exactly what we're looking for. For this reason, we generally seek to think about the funders who seem like the best fit to support a given project, then try to understand why they aren't supporting it. Sometimes the reasons turn out to be compelling to us, and sometimes they end up seeming like reasons we are happy to discount (particularly when the reasons are "structural," i.e., a project is considered to be out of another funder’s focus areas rather than having anything wrong with it).

We seek to maintain contact with other funders who work on similar causes to us. One benefit of doing so is that we can learn from them. Another is that we can avoid doing work that is redundant with theirs. Having a practice of trying to understand why they don't support the things we support, as described in the above paragraph, accomplishes some of each.

Another benefit is that we can sometimes pitch promising giving opportunities to them, which can result in the projects we find most exciting getting more support. In some cases, another funder might be excited to commit all the funding that is needed, making our own support for a project unnecessary. We've also heard informally that many funders value "getting in on the ground floor," so involving another funder early on may make a big difference to a project's or organization's long-term funding situation.

But the kind of informal contact described above – checking in, discussing potential grants, pitching giving opportunities – falls well short of formal, large-scale, or long-term collaborations. Our feeling is that a collaboration of this sort with a staffed foundation – possessing its own focus areas, strategy and style – would likely introduce a significant amount of time cost and coordination challenges, and we haven't yet come across a situation in which that seemed like the best approach.

How should we evaluate the results of our grants?

Out of all the questions we've listed, this may be the one that has been most extensively discussed in writing:

• Money Well Spent, co-authored by Paul Brest (former President of the William and Flora Hewlett Foundation), is a guide to "strategic philanthropy." It emphasizes setting out clear goals, success indicators, and milestones up front. As a Hewlett grantee, we have generally been asked to set quantifiable goals each year, while reporting on progress against last year's goals.
• Bill Schambra has written counterpoints to this philosophy (example) stressing the importance of local knowledge. He generally encourages funders to give in their local communities, and to "Go down to the neighborhood and check it out, armed with common sense."
• The Elusive Craft of Evaluating Advocacy gives multiple reasons for the difficulty of applying a "strategic philanthropy" approach to policy-oriented philanthropy. (We have previously written about this essay.) Its recommendations:

  The best way to evaluate an organization whose influence is extremely diffuse is for grant officers to be close to the political action and thus able to make informed judgment calls on how it conducts its core activities ... Equally important is an organization’s strategic capacity, which can be defined not only as its formal strategic plan, or the wisdom of its senior leadership (two factors that funders tend to focus on), but also the organization’s overall ability to think and act collectively, and adapt to opportunities and challenges ... Yet another way to measure an organization’s quality and influence is through “network evaluation”—figuring out its reputation and influence in its policy space.

We believe the best approach to evaluation depends very much on the specific case. When evaluating GiveWell top charities – which are defined by relatively linear, quantifiable, evidence-backed uses of money – we set clear expectations and assess progress regularly, along "strategic philanthropy" lines. But we don't think this approach is appropriate in all cases, particularly when working on very long time horizons and on high-risk goals, such that the points raised in "The Elusive Craft of Evaluating Advocacy" become more relevant. In general, our writeups on each grant include discussion of our plans for follow-up.