The what, why, and how of impact assessments

What is an impact assessment?

An impact assessment is a research study that measures how the services of a microfinance institution (MFI) impact the lives of its clients in such areas as employment, income, nutrition, education, health, and gender equity. It is the primary tool used to determine the effectiveness of microfinance as a development intervention.

How are impact assessments used?

MFIs use impact assessments to determine whether their services are having a positive impact on the poor clients they seek to serve. At the very least, they can use an assessment to establish whether their services are producing any negative impacts (e.g., growing client indebtedness) and take action to correct such effects.

Significantly, MFIs can also use impact assessments as a management tool to improve operational efficiency, product design, and social effectiveness. Impact assessments encourage an MFI to include impact data in its information systems and decision making. Regular collection and analysis of impact data enables an MFI to better identify and serve the needs of its clients, and thus function more effectively.

Impact assessments can serve a variety of objectives, including internal learning for MFI staff, improving operational efficiency, enhancing management and information systems, or proving the positive impact of financial services to external donors. Assessments vary depending on the type of microfinance institution, its mission, the objectives it seeks to accomplish with an assessment, the type of information required, and cost considerations.

Why are impact assessments important?

by Anton Simanowitz, Coordinator, Imp-Act Programme,
Institute of Development Studies, University of Sussex

Microfinance programmes are one of the most important interventions to reduce poverty in developing countries. Recent years have seen huge growth in the number and size of microfinance organisations, the volume of microfinance clients, and the provision of subsidised donor funding. Poverty reduction is part of the mission of a large proportion of microfinance providers, and donor funding is allocated to microfinance on this basis. There is, therefore, a basic need to understand and improve the impact of microfinance institutions (MFIs) as a key premise of successful poverty reduction.

Most MFIs do not, however, measure the impact of their work, nor do they seek to learn ways in which they can improve their impact on the poor clients they seek to serve. In fact, some MFIs suggest that financial performance indicators are sufficient to indicate whether or not they are doing a good job, arguing that if clients are willing to pay for a service (i.e., an institution's client retention and repayment rates are good), it can be assumed that they are happy to pay for the MFI's services because they are doing them good. From this point of view, the market is the indicator of impact.

But financial performance does not measure change in people's lives. Indebted clients may repay loans even when their businesses fail and much hardship results. Market proxies mask the range of client responses to and benefits of the financial services offered by MFIs. Indeed, the complexity of impact assessment stems from the numerous interrelated factors that influence client decisions and actions and thus affect the effectiveness of the organisation in achieving positive impacts.

It is critical for MFIs, particularly those working with very poor and vulnerable clients, to be sensitive to the impacts of their work?particularly to the potential negative impacts that their services may produce. An MFI should know that it is having a positive impact or, at the very least, that it is not having a negative impact on some people. See the Practitioner Pages to read how and why MFIs have integrated impact assessment into their business.

Impact Assessment is for Practitioners and Clients, not just Donors

All MFIs have knowledge about how their work affects their clients, but this knowledge is often not formalised. Many organisations, for example, use market research to understand client needs and reactions, and develop management information systems that can detect problems in operations or methodology.

A great deal of impact assessment work conducted in the past, however, has been donor driven and mostly serves donor needs. The emphasis of these assessments has been on justifying funding, rather than helping MFI management to learn and improve their work. Increasingly, however, a more "practitioner-oriented" approach to impact assessment is emerging. This approach seeks to:

• fit into existing MFI work patterns and provide frequent and timely information

• learn from and build on existing knowledge and experience of an MFI

• produce results that can be easily used by MFI management

• feed into management and product design processes, allowing MFIs to improve their services and, thus, their impact on clients

Most MFIs seek to meet the needs of multiple stakeholders and to fulfill a number of different objectives, including the need for credible information to influence donors and other external stakeholders. Most impact assessments will therefore include a mix of objectives and approaches.

The Impact Assessment Centre provides guidance and information about how impact assessments can better answer the needs of MFIs, i.e., how they can be designed to provide credible information to improve MFI operational practices. It also considers how to design studies and systems that balance the needs and objectives of different stakeholders. For more detailed information, see Developing an Impact Assessment.

Credible Impact Assessment

Impact assessment is hard to do well. The key is to provide information that is sufficiently credible for the specific purpose of the assessment. Practitioners do not need information that is 99.9 percent accurate. They can accept a margin of error, provided the information is credible and can assist management in operational decisions and product design. An emphasis on timely and useful results leads to impact assessments that are more cost-effective and simpler than donor studies, which have attempted to demonstrate causality.

A considerable amount of debate has centered on the question of causality and attribution in microfinance impact assessment, that is, whether it is possible to demonstrate that the particular intervention of an MFI has led to a specific change in a client?s life. Client livelihoods and the communities in which they live are complex. Clients, for example, may have multiple income sources. The credit provided by an MFI is, moreover, fungible and not necessarily used for the purpose for which it was requested. These conditions make it difficult to attribute specific impacts to an MFI. The difficulty of impact attribution is exacerbated by the challenge of establishing effective control-group mechanisms that can create a counter-factual (i.e., what would have happened to a potential client without the MFI?s intervention). The methods that can be used to establish such a counter-factual are, moreover, time-consuming, costly, and complex.

Even a low-cost impact assessment needs to be credible. It is important that the assessment process is carefully thought through. Indicators must be developed based on a sound conceptual framework for the type of impacts being examined. Systems used to collect information need to be effective and produce reliable results. Analysis needs to be carefully planned to ensure that results will provide the right type of information. Using a mix of methods and triangulating information from one method against another is a crucial technique that can both improve the credibility of results and lower the costs of an impact assessment.

The Impact Assessment Centre aims to provide a practical guide to enable people to develop impact assessment processes and systems that are appropriate to the context, objectives, and stakeholder needs of their organisation. An appropriate system is one that provides information that is credible and sufficiently reliable for the specific objectives of an individual MFI.

Anton Simanowitz

Coordinator, Imp-Act Programme,

Institute of Development Studies

University of Sussex