Assuring Quality

And capturing impact in market
development programs

CAVAC has invested great resources in developing a modern and effective monitoring and evaluation (M&E) system that is tailored to market development programs. CAVAC has a system that works and is on track. Not only is it improving the quality of program initiatives, but it is also producing reliable impact data.

Investing in M&E

For decades, numerous programs have promoted economic growth yet few have been able to supply reliable impact data. CAVAC has set up a monitoring and evaluation (M&E) system to deliver such data whilst helping to manage and keep its initiatives on track.

There are three main reasons to invest in a system of monitoring, assessing change and capturing impact:

Demonstrating impact

  • Especially in the last decade, donor governments and their citizens have become more critical about how development money is spent and increasingly demand that programs show impact. Nevertheless, few market development programs can provide solid impact data.

Adjusting and steering

  • Market development programs that seek to achieve change in complex socio-economic realities cannot simply execute a plan. Success depends on more than just technical solutions. It also depends on how well and how quickly farmers or companies innovate. It is difficult to predict what will happen when programs interact with a target group indirectly by stimulating public or private institutions to better interact with farmers as CAVAC does. Since things never go completely to plan there is always a need to monitor, rethink what is needed and adjust the activities.

Making portfolio decisions

  • Unfortunately, achieving impact takes time, sometimes many years. A system that considers the potential impact of an initiative at an early stage of activity development is supportive to the design process. Such a system allows the program to communicate expected impact at a much early stage than conventional systems.

CAVAC'S M&E SYSTEM

MB-6-1

If M&E was easy most programs would provide reliable data, unfortunately for most programs this is not the case. Monitoring experts have tried setting up systems where external, independent experts measure impact based on quasi-experimental designs. When this approach works it can generate reliable data for verifying impact. Whilst this approach may be the best tool for relatively simple interventions that are well-defined at the design phase, it does not cater to the needs of an integrated project M&E system especially if the activities change continuously. The nature of interventions often make it almost impossible to construct a reliable design when it is unclear who the specific beneficiaries are, how they innovate and how much the control group is affected by other information.

A quasi-experimental design with control groups may serve to verify specific impacts of particular activities, but as an integrated system it does not enable the functionality to steer or improve implementation as is required by the Program. By the time data shows that the expected impact did not take place it may be too late to adjust. In programs such as CAVAC where the staff continually interact with project partners to observe changes and steer outcomes, while only partially capturing this detail in program reporting, it is difficult for independent “outsiders” to assess and quantify what is really happening.

full No image

In recent years, a number of programs have started to use an internal M&E system based on result chains or impact logics. Impact logics are a good tool to:

  • Improve designs of activities;
  • Monitor progress of intermediate changes and final impacts;
  • Build a management system that allows for continuous steering; and
  • Create a framework for collecting reliable impact data that can be plausibly connected to program activities.

CAVAC’s M&E system is based on use of of result chains or impact logics. By having a number of common key performance indicators in all impact logics CAVAC is able to aggregate data on specific indicators, such as outreach, yields, areas under cultivation and quality, and report on the overall impact. A system of impact logics also allows for great flexibility in balancing credibility with costs and efficiency.

A system based on impact logics works well if implemented by the same people who are responsible for day-to-day activities. The Donor Committee for Enterprise Development (DCED) has developed with a quality system to assess the way a program executes its M&E system. Though an external impact assessment would be more credible, an internal system with an external quality check may be the most feasible choice.

CAVAC has set up and is currently implementing a system whereby CAVAC Technical Experts and Monitoring Specialists jointly draft an impact-logic for every initiative. Each impact logic describes key steps and for each step from inception to final impact at least one indicator is chosen. A monitoring plan is then drafted that describes how and when each indicator is to be measured. Some indicators are critical for getting credible data and may require large surveys, while other indicators require only simple observations or smaller-scale surveys. Both CAVAC’s Technical Experts and the M&E Specialists share the tasks of monitoring the indicators.

Regular Reviews

At regular intervals each implementing team sits together for at least one day to review the results of their initiatives. Each team reviews the M&E studies, reflects on their daily observations and assesses whether the initiatives are on track and where remedial action may be needed. In many cases the teams realize that they need more information to understand why things happen or do not happen as planned. In these cases new studies are designed. When monitoring is carried out as planned and intermediate and final indicators are measured and evaluated the impact of the program gradually emerges.

CAVAC is on track with its M&E system: all initiatives have impact logics, are monitored continuously and each quarter CAVAC conducts reviews that lead to adjustments in the initiatives. The reviews also show what new activities CAVAC should focus on or what activities should be stopped prematurely due to lack of progress.

Expertise

CAVAC’s Technical Experts have been selected for their ability to analyse economic changes and to understand the behaviour of entrepreneurs, companies, government, farmers and farmer organizations. CAVAC also has two dedicated M&E Specialists who maintain the system and ensure that all impact logics and monitoring plans are updated. The M&E Specialists also take the lead in conducting or outsourcing larger surveys and studies and then help the Experts to interpret the findings.

Most staff need to spend at least 10% of their time on M&E. This should be seen not as down time or a program cost but rather as an investment in ‘doing the right thing’ to ensure impact.

Cultivating Openness and Reflection

An M&E system where national experts collect much of the data and interpret the lessons can only work when they are convinced that the truth is important, even when it means revealing failures. A program that chooses such a system therefore needs a culture of openness and honesty and of critical reflection where failed initiatives are not seen as personal failure but as a normal part of project business.

As obvious as this may sound, it can be difficult to induce and maintain such culture. Most organisations expect loyalty which often means that staff keep failures to themselves. Many donors also find it hard to rationalise when a few things go wrong and find it difficult to understand that market development programs take risks and that means that some activities will not be successful.

Cultivating and maintaining this culture of openness and honesty has been a very deliberate effort from the beginning of CAVAC. Management is very committed to continuously reinforcing this message, regarding honesty and discouraging covering up failures. CAVAC has been successful in this to some degree.

To sustain this open and critical culture requires proactive measures. For example, M&E can reveal when remedial actions are needed. However, Specialists will often be reluctant to continue raising concerns if the Program is not able to react quickly. Such flexibility requires entrusting Specialists, who do the day-to-day work with the authority to make decisions. It also requires Managers to take the time to listen to concerns and help devise solutions.

Importantly it requires systems that allow for rapid, inevitable changes. Unfortunately, program systems are normally built on pre-approval and based on government rules that are by nature bureaucratic. The donor, DFAT, and the implementing company, Cardno Emerging Markets, have tried hard to create enough flexibility to ensure responsive action is possible however at times this can be difficult.

Impact Takes Time

Delivering and measuring impact takes time, especially when a program does not support farmers directly but improves the markets that serve them. It also takes time for farmers to innovate. Most farmers first want to see if their neighbours succeed before testing innovations themselves in the next cropping season which is almost always a year later. One or two years after starting an activity, changes in the support markets will become more visible but final impacts on farmers’ incomes can take 5 years or more. Cardno Emerging Markets and DFAT agreed early on when it would be feasible to release different kinds of impact data and have kept to this schedule.

External Quality Control

CAVAC’s M&E system is designed in line with the DCED quality standard. CAVAC conducted an early internal audit to check compliance. In 2013, CAVAC commissioned a  DCED Standard’s approved auditor to conduct a compliance assessment of its M&E system. The audit scope covered the entire CAVAC portfolio, except for the BEE and Vegetable sectors.

Reporting

CAVAC’s M&E system is an effective tool to improve and steer projects. It also has all elements in place to collect all types of impact data over time. Bi-annual reports to DFAT and the Royal Government of Cambodia (RGC) provide updates on major activities, progress on the key indicators and lessons learned. To explain the activities and impacts better CAVAC also produces small case studies on a bi-annual basis. The RGC has also requested indicators that monitor progress on initiatives. CAVAC produces a report for the Government each month and quarterly reports cover operational milestones set annually.

For each initiative, CAVAC captures outreach data (number of farmers affected), assesses sustainability and at least one of the following three overall indicators: increase in yields, additional area under cultivation and quality. Except for sustainability these indicators can be aggregated to give overall program impact data. With this data, based upon a number of assumptions, CAVAC can calculate what additional income farmers will have and how much more rice and vegetables they will produce due to CAVAC’s activities. With the overall indicators, CAVAC can also report on a number of headline indicators that DFAT has developed to monitor all of its programs. The program will end in December 2015 and it will report on outreach and impacts until 2017.

Data by gender and people living with a disability

CAVAC monitors, as far as feasible, how many men and women take part in CAVAC-supported activities. Separating final beneficiaries by gender, however, would not accurately reflect reality. In most cases, farming is a household activity that men and women jointly undertake and make decisions on, even if some specific activities are done mainly or even solely by men or women. Therefore, CAVAC initiatives normally do not benefit farmers of one gender to the exclusion of the other. Instead CAVAC has developed a system of household typologies that shows who takes decisions and who undertakes the activities for most markets that benefit from CAVAC initiatives. A similar approach is applied to capture the impact on people living with a disability. For more information on the CAVAC approach to gender and people living with a disability please see the CAVAC website.

Data Integrity

CAVAC is fully committed to publishing data that is credible and specifically attributable to CAVAC’s activities. For many of CAVAC’s initiatives this is possible, however, for some it is a stretch. For example: CAVAC helps initiate a television program that is likely to lead to improvements in farming and therefore increased yields. Capturing these changes and showing that they happen because of CAVAC is too difficult to be credible. CAVAC therefore ignores some of its impacts resulting in inevitable underreporting. CAVAC activities also result in a number of unintended impacts. These are also excluded in its formal M&E reporting.

Designing an M&E system requires many decisions about what to include or exclude as well as about timing. Major gains in efficiency can also be made by assuming a number of things rather than measuring them. For example: using an average rather than measuring each case individually can save resources.

Four rules govern CAVAC’s assumptions; they must:

  1. Comply with the DCED Standard,
  2. Be transparent so that outsiders can easily check their credibility,
  3. Be conservative; and
  4. Be based on credible literature, if at all possible.

Summary

With the demand for reliable impact data growing, CAVAC applies an M&E approach based on continuous and flexible monitoring of results in the field. The system uses impact logics while building in quasi-experimental designs to measure some individual indicators. A key success factor of CAVAC’s approach is its deliberate openness, creating a professional culture that proactively encourages honesty, even when reporting negative results. The combination of these elements ensures that CAVAC has what it needs to improve the quality of its initiatives whilst producing reliable impact data.

Further reading

  • CAVAC’s M&E manual (this can be downloaded from the Further Reading section of website) gives detailed overview of how CAVAC implements M&E.
  • The DCED website has a vast amount of material covering M&E approaches and project experience/approaches.

Appendix 1

Additional Technical Aspects

  • Displacement is corrected for if it is expected to have a serious impact and is related to farmers. Displacement of support providers is considered but not always seen as negative if the overall quality and availability of the service improves. In general, CAVAC chooses initiatives that create markets and avoids situations that displace market players.
  • ‘Crowding in’ is an essential part of market development and as such is captured in every impact logic. The impact, however, is expected to be minimal during the period that CAVAC monitors.
  • CAVAC separates its impact as either direct or indirect such as impact that results from crowding in or copying.
  • CAVAC only captures impact on the production and incomes of farmers’ households. It does not capture its impact on other players in the value chain.
  • Instead of measuring yields and income, CAVAC calculates these impacts from measuring changes in farmer practices. Though this may sound counter-intuitive CAVAC is convinced it is often a more accurate way of attributing impact.

Appendix 2

An Example of Impact Logic and Monitoring Plan

No Image

 

Gender survey result
  Mainly men Mainly women* Joint
Decision 40% 30% 30%
Purchase 60% 40%  
Application 90% 10%  
Decision making base on rice crop pattern
Wet season rice only 30% 40% 30%
Others(dry, early wet) 40% 30% 30%
 
Economic status: 14% relatively poor as the households own not more than 0.5 ha of land, have no motorbike, nor TV.
 
* The % has not included female widowers who constitute around 10% of farming households.

 

Business case

1st IP: CAVAC provides capacity building to extension and technical staff of Nokor Thom company so that they can improve their extension services especially their Field Demonstration (FD), emergency interventions (EI), door to door extension activities (DD) (also PDA staff do extension for the company in other projects or PDA work they involve to farmers). Providing capacity building to field staff would enable the company to properly provide effective extension services to farmers through their existing extension program. Improved extension will enable farmers to increase their yield through more appropriate insecticide/herbicide applications. Scientific research indicates that a yield increase of up to 19% is possible while CAVAC’s own research indicates that weed control could significantly benefit yields of dry season rice production

2nd IP: CAVAC supports Nokorthom Company to design a clear Extension Strategy. By so doing, the company would better operate their extension in term of outreach and quality. Farmers will be able to increase their yield through the this quality extension service on pest control. An international consultant will be recruited to work on this assignment. With the new extension strategy, Nokorthom Company would be able to strengthen the quality in current target location and expansion to other potential location.

 

Projection/YearCAVACNon-CAVACTotal
  area Indirect  
Nokor Thom staff trained -8-
PDA trained 3 12  
# of farmers reached by FD 450 1,800  
# of farmers reached by EI 2,274 9,096  
# of farmers reached by DD 900 3,600  
Total 3,624 14,496  
80% of farmers get knowledge 2,900 11,600 14,500
80% of farmers apply 2,300 9,300 11,600
75% of farmers increase yield 1,700 7,000 8,700
 
The projecting number will be validate

 

Assumption

Box 10:IL will be updated the box if the company sets up new extension activity.

 

 

 

 

CAVAC: technical brief on Assuring quality