Donate

Large-scale PIMS in Bangladesh: using data to unravel the sustainability enigma

By Daniel Meeson, University of York

WASH Matters

12 Sep 2014

During the next two months, WaterAid Bangladesh will be visiting and studying about 1,800 waterpoints, 3,000 households and 400 sanitation interventions across the country to form one of the largest contributions to our sector-leading post-implementation monitoring surveys (PIMS). The purpose of such a huge exercise is vitally important – by systematically measuring functionality across time we can begin to reveal the elusive nature of sustainability.

PIMS is a central part of WaterAid’s monitoring and evaluation toolkit and an annual requirement for each of our country programmes. It comes in two sizes: small-scale and large-scale. The small-scale type is an opportunity for focused, selective study, whereas the large-scale version has the ambitious aim of getting good estimates across our projects in a particular country. This investment is at the heart of the sector-wide focus on sustainability, because by measuring functionality in all its guises – against age of construction, region, partner, technology, user numbers, support in repair and maintenance etc. – we get a picture of where and why interventions are succeeding or failing. This level of detail is essential not only to advise policy, but also as a system of indicators that demand action now.

Large-scale challenges

Helping to design Bangladesh’s large-scale survey has been a challenging, technical and exciting task – for this mathematics student at least! Even if one manages to avoid bias and error, the challenge is to navigate a path between cost and logistics to reach the goal: precise and representative estimates of WaterAid Bangladesh’s contribution to the water, sanitation and hygiene (WASH) sector. This contribution was around 60,000 constructions and renovations in the past ten years alone. Statistics ensure that our relatively small sample can reliably estimate the state of the entire country programme – what we mean by the often-misused concept of statistical significance.

The large size of Bangladesh led us to reflect on our standard methods. Finding the need to improve logistically, we delved deep into sampling theory and compared our methods with those of UNICEF’s rapid assessments of drinking water quality, the UN’s multiple indicator cluster surveys and other systems. The final design was a self-weighting multistage survey, stratified for district (zila) and sub-district (upazila) representation, followed by cluster sampling across the unions (which each contain about nine villages). The waterpoints, homes and latrines that the teams will visit are in the 51 unions we selected. Coping with a patchy and varied sampling frame was a real challenge, because hugely different union sizes meant much careful adaptation was needed.

Of course, sampling strategy and methodology are not the full story. The surprisingly tricky task of even deciding what questions to ask required much thought, especially as PIMS reaches across the entire spectrum of indicators. Here the process benefits from being in its fourth year of development, with each iteration having provided a better focused and more informative set of questionnaires.

Technological help

New to the 2014 PIMS (in Malawi as well as Bangladesh) is our innovative partnership with mWater and its mobile data-collection platform. Technology-driven surveying allows for improved consistency and ease of recording results, adaptable language capabilities, and, most significantly, waterpoint mapping. The change from handwritten coordinates (based on various systems) to a consistent, automatic entry system will simplify the mapping process and speed up our contribution to this exciting movement. Finally, and no less significantly, mobile data entry will put an end to the mountains of paperwork and the chore of data entry for both entrant and analyst.

It must be stressed, however, that PIMS is not merely a number-crunching, bean-counting exercise, but a means to an end – the end being to convince and support providers and governments towards achieving truly sustainable development.

The size and scope of the PIMS process make for powerful and interactive visualisations, which are fascinating for policy-makers and hopefully, to some extent, for the public too. Each year, the results are reinforced and the conclusions grow stronger; as the process matures, comparisons at the global level (and with other institutions) will be especially revealing.

Sharing opinions

This year’s analysis in Bangladesh will benefit from sharing of results with in-country partners and asking for their opinions directly. Engaging those involved in addressing questions such as why interventions in certain regions have been weaker than in others, why certain technologies have lasted longer than others, or how different community management techniques contributed (or did not contribute) to reducing open defecation rates. Such conversations will contribute to the overall PIMS picture, complementing solid statistics with more qualitative insights.

Nevertheless, we must always keep in mind the limitations of surveying. Statistical snapshots on their own give few details of the programmatic aims or techniques behind an intervention, and estimates are only that – estimates. However, armed with this awareness, and supported by thorough analysis, PIMS will prove invaluable to the WASH sector’s quest to find what works and how to keep it working.

Notes:

  • By early 2015, WaterAid will be regularly publishing the analysis and interpretations of results from all PIMS exercises on its website.
  • Daniel Meeson spent two months with the Programme Support Unit in London, and is entering his fourth and final year of a Master of Mathematics degree at the University of York. He would like to thank Mike Smith, Arjen Naafs, Petri Autio and Ruth Hinds for their great support and encouragement during the project.

1 Bostoen, Kristof (2007) Measuring access and practice: designing a survey methodology for the hygiene, sanitation and water sector, PhD thesis, London School of Hygiene Tropical Medicine.