A conversation with Gayle Martin of the World Bank on July 30, 2014

Participants

Note: These notes were compiled by GiveWell and give an overview of the major points made by Dr. Gayle Martin.

Summary

GiveWell and Good Ventures spoke with Dr. Gayle Martin of the World Bank to receive an update on the Service Delivery Indicators (SDI) program, to which Good Ventures made a grant of $500,000 (http://www.goodventures.org/our-portfolio/grants/service-delivery-indicators-project-march-2014) in March 2014. Conversation topics included SDI’s progress over the last year, SDI’s goals for 2015, and SDI’s funding gaps.

SDI’s progress

SDI is a program that collects information on health and education services at the country level. Its indicators measure the availability of important health and education inputs as well as the knowledge and effort of the service providers. SDI has already gathered and analyzed data for six countries:

The fieldwork of gathering data has recently been completed in two more countries:

SDI is almost ready to collect a first round of data in several more countries:

Examples of impact

SDI’s data has already started to have an impact. For example, in Tanzania, the Prime Minister launched a national initiative called “Big Results Now” in an attempt to improve the country’s health and education services. A lot of money had been spent on education in the preceding decade, but results were unimpressive. However, by focusing on quality data and the incentives for service providers—a key focus of “Big Results Now”—Dr. Martin believes that SDI’s data has now helped to explain the gap between the government’s efforts and the poor health and education outcomes. SDI is pleased that its data was available when the Tanzanian government needed it, and believes that the government has found its explanations to be useful.

Another way in which SDI is having an impact is by generating public discussion. For example, when the information from SDI’s surveys was released publicly in Uganda, it sparked a heated debate about why the education results were very poor. Additionally, teachers were on strike in both Uganda and Kenya when SDI’s data was released in those countries, and when the survey results were released they impacted the discussions around the strikes. Furthermore, articles in high-profile news sources (e.g., the Economist and BBC) are beginning to discuss the data that SDI is collecting. SDI is not mentioned in these articles explicitly, but it is the only possible source of the information discussed in the articles. SDI feels it is useful to add concrete data to public discourse and considers these public debates to be successes.

Finally, the World Bank has begun to use SDI’s indicators as some of its disbursement-linked indicators. Disbursement-linked indicators are sometimes used when the World Bank makes a conditional funding agreement with a country. Essentially, the World Bank will agree to fund a country’s program if the program is shown to be achieving certain performance targets. Disbursement-linked indicators are used to show whether or not a program is achieving its performance targets. In both Tanzania and Uganda, the World Bank is now using SDI’s data as a disbursement-linked indicator. There is also a risk: this could end up making SDI’s task more difficult, because if $10-15 million is dependent on the outcomes of SDI’s surveys, that could incentivize people to inflate the countries’ performance metrics. This risk will have to be carefully tracked to avoid unintended consequences, meaning SDI may have to increase its quality-assurance efforts.

SDI’s goals

In line with its original objectives, SDI hopes to complete 10-15 countries by the end of 2015. Eventually, SDI hopes to create a brand for itself and encourage expectations that its data should be available for every country.

Repeatable surveys

SDI expects to have 2-3 repeated surveys by the end of 2015 (surveys that have been done for the second time in a country previously surveyed). Repeated surveys are very important to SDI. One survey can reveal the current state of a country’s health or education services, and often shows that these services are as low-quality as expected or worse. However, it is more important to know whether or not those services are improving; that knowledge requires repeated surveys. Dr. Martin believes that countries and donors sometimes use money inefficiently by investing in one-time data collection efforts. If governments chose to focus on systematic data collection instead, then perhaps some funds going to one-time surveys could instead go to repeatable surveys.

SDI hopes to help governments improve their services and thus tries to focus on the question, “Are the health and education services in this country improving?” This question is helpful for engaging governments, since it does not criticize countries for having subpar services (such criticism can put governments on the defensive). Also, by focusing on this question of improvement via its repeated surveys, SDI hopes to convince new countries that its surveys are not intended to show how badly a country is doing, but are rather intended to measure how services are changing over time.

Challenges

There are a few challenges that SDI has experienced in its early stages. One challenge is that fieldwork is more difficult in some countries than others. For example, surveys in Mozambique and the Democratic Republic of the Congo are challenging. However, a large number of people’s service experiences can be covered by gathering data in the DRC, so SDI considers the potential outcomes to be worth these challenges.

SDI has also experienced some funding challenges. Although it originally predicted $400k-$500k for 2 surveys (health and education) per country, actual costs have been closer to $700k per country (this covers both the health and the education surveys). In part this is because contracts with survey implementers alone cost about $500k. SDI also underestimated the cost of additional oversight for quality assurance and the costs of dissemination of findings. SDI believes that there are economies of scale in surveying countries, and that once it scales up its activities, some inefficiency can be reduced. However, these potential savings should not be overstated.

Changes in SDI’s indicators

SDI has had to “tweak” a few aspects of its measurements for practical reasons, but there have not been any changes large enough to be considered a departure from what SDI originally planned to measure. These tweaks occurred in the first four countries SDI surveyed. SDI is not worried about these tweaks; it wants its surveys to be strong instruments that can be repeated every 2-3 years and that goal requires some initial adjustments. Note these are tweaks but not substantial additions. Because SDI values repeatable surveys, it considers the burden of its surveys when making additions to the survey instrument. If the survey becomes too burdensome on participants it will become untenable.

Funding gaps

SDI is currently not fully funded for all 15 countries that it hopes to reach by the end of 2015. It is underfunded in about 5-6 countries, so its immediate funding gap is roughly $4 million. It would prefer to have this funding by December 2014 or January 2015 in order to start the process of surveying in these countries. The latest that it could receive the funding would be February 2015.

SDI’s total budget is $700k per country, although it was slightly higher in the first few years. In Africa, once SDI is working in 20-25 countries, it will have covered all of the countries where the political environment and public domain is sufficiently open to use its information. Essentially, there would be no need to expand to more countries in Africa. Beyond 2015, there is interest from other regions, which SDI will have to consider.

Current funding strategy

So far, SDI has been opportunistic about leveraging sources of funding within the countries it is surveying to complement the resources in the multi-donor trust fund resources. For example, it has used funds from the Department for International Development (DFID) and the Gates Foundation (in Nigeria) on a country by country basis. This is not SDI’s preferred model of funding because it elevates transaction costs—costs would be fewer if the funds were all in one pool. In discussions 2012/13 DFID has told SDI that it did not have a lot of money to put into a multi-lateral project, but it has been buying into SDI’s activities in countries where there are DFID offices (e.g. Tanzania, Zambia, and Mozambique).

Given the funding constraints SDI has preferred to leverage resources at the country level, including World Bank resources, so that its multi-donor trust fund resources last longer. However, the trust money is still being spent to supplement the in-country funds.

Because its current funding strategy relies on complementary country-level funds, SDI has prioritized countries with such funds.

Impact of a lack of funds

If SDI is unable to get the funds it is hoping for, it will survey fewer countries, rather than decreasing the budget per country. Scaling down the per-country budget would require SDI to decrease activities like quality assurance and dissemination, which are essential to the project’s success. SDI believes that it would be easier to scale up the number of countries later than to increase the quality of its surveys later.

Financial sustainability

The future funding model has implications for transparency. A model where funding is exclusively coming from one particular party – whether it be governments or the World Bank – has risks for transparency.

Furthermore, information is a global public good and a joint government-development partner funding model will be optimal

Sustainable implementation

A key principle is that the SDI implementation model relies on is the use of national research institutes as implementing partners. Continuing this model will be critical since countries benefit from the capacity built through implementing the SDI surveys. Furthermore, because the surveys implemented in this manner would end up bringing money back into the country through the salaries of the researchers, funders may be more likely to support the program.

The World Bank’s resources

The World Bank is currently going through a reorganization, which has led to budget cuts. For example, within the Africa region there used to be a director for human development who oversaw three sectors, including health and education. This was Dr. Martin’s previous supervisor, but now that position no longer exists.

Currently, the World Bank has two separate sectors for health and education, and SDI has staff in each of these areas. This presents challenges for SDI; previously, it was helpful to have an office that oversaw and understood both sectors. Now that the sectors are managed separately, it could be problematic if, for example, SDI employees were rewarded for their work in one sector more than another (e.g. health instead of education). Currently it seems that both sectors are being rewarded equally. SDI is currently exploring its options for long-term sustainability within the bank.

Despite the changes, SDI is not worried about losing the Bank’s support. Whenever the SDI program is presented to Bank leadership they are enthusiastic about the project. There is a lot of interest in cross-sectoral work at the World Bank. In fact, the performance agreements of directors require that they demonstrate their commitment to cross-sectoral work. So, SDI can be an example of cross-sectoral work in the portfolios for health and education directors.

These factors contribute to SDI’s optimism about long-term funding possibilities. For example, there is likely room to leverage the World Bank’s resources. SDI will probably never be fully funded by the World Bank, but last fiscal year it received roughly $500k from individual country directors who supported SDI. So, there is potential for the Bank to prioritize SDI as a source of information it should support with dedicated funding. The World Bank already invests in information on poverty through household surveys, indicating its capacity for sustained commitment to survey information.