Millennium Villages Project continues to systematically overstate its effects

by Michael Clemens and Gabriel Demombynes

The Millennium Villages Project (MVP) is an experimental anti-poverty interventionin villages across Africa. In October, we released evidence that the Project’s official publications were overstating its real effects, and we offered suggestions on improving its impact evaluation. On Tuesday the MVP, whose leadership and staff are aware of our work, continued to greatly overstate its impact.

It started last year. In a June 2010 report called Harvests of Development, the Project claimed that the impacts of the project included expanded cell phone ownership.  For example, the MVP claimed that increases in cell phone ownership at the Ghana site were caused by the project, in this extract from page 91 of the MVP report:

This claim has little basis, because cell phone ownership has been expanding at about the same rate all around the MVP site in areas untouched by the project. The graph below, from our paper, shows cell phone ownership at the MVP site in black compared with various other areas:

No reasonable person could look at these findings and conclude that the MVP intervention clearly caused any substantial increase in the rate of expansion of cell phone ownership. The evidence gives no clear reason to believe that cell phone ownership would have expanded any more slowly at the intervention sites if the Project had never existed.

But on Tuesday, months after multiple discussions we’ve had with MVP leaders on our research, a post on the MVP’s blog restated the claim that the increase in mobile phone ownership at the intervention sites was caused by the Project, calling the increase at the Sauri, Kenya site one of the MVP’s “achievements.”

As the World Bank Chief Economist for Africa Shanta Devarajan has observed, our evidence does suggest that the MVP has had some positive short-run impacts on people’s lives. So there is no reason to overstate the impacts.

Before-vs.-after comparisons implicitly assume that in the absence of the intervention, nothing would have changed. This assumption is demonstrably incorrect in the case of the MVP and cell phone ownership. The cell phones are just one example of how the Project overstates its impacts; in the paper we discuss others. For a brief summary and a discussion of my visit to the Sauri MVP site, see this earlier post

http://blogs.worldbank.org/africacan/millennium-villages-project-continues-to-systematically-overstate-its-effects

Manual/Toolkit – Making Aid Work: Towards Better Development Results Practical guidance for parliamentarians on the role of p.

Alan Hudson; Leni Wild; Julia Weinstock;
March 2010

Parliaments and parliamentarians have a crucial role to play in ensuring that governments are accountable for the decisions that they make about how resources – including aid – are spent. The scope parliaments actually have to play this role varies widely. Some parliaments benefit from large resources and a legal framework that back them in playing their oversight and legislative role. Many other parliaments, especially in developing countries, lack resources or power to play an effective
role in promoting development or the more effective use of aid. Parliamentarians themselves come from all walks of life and do not share the same knowledge on these issues, and there is no consensus among parliamentarians or across countries on the ways and means by which they can enhance oversight of development policies and how development resources are used.
This Guidance Note addresses some of these challenges and seeks to provide parliamentarians and those who work with them with a common understanding and clear guidance on what they can do to promote more effective and accountable use of aid in particular and of development resources in general. It is especially relevant for countries in which aid forms a significant share of total development resources, and where parliaments and parliamentarians must be involved in the discussion of aid resources. It forms part of the Capacity Development for Development Effectiveness (CDDE) Facility’s work plan and it involved a joint IPU-CDDE partnership, with inputs from ODI researchers.

Published by Inter-Parliamentary Union.

http://www.odi.org.uk/resources/details.asp?id=4799&title=making-aid-work-practical-guidance-parliamentarians

course from MIT’s Jameel Poverty Action Lab (J-PAL), “Evaluating Social Programs”

Dear all,

I wanted to bring your attention to the 5-day Executive Education course from MIT’s Jameel Poverty Action Lab (J-PAL), “Evaluating Social Programs”. Courses will be held in Cambridge, MA (USA); Rabat, Morocco; Bogota, Colombia; and Chennai, India.

Information on content, dates, and the application process can be found here:
http://www.povertyactionlab.com/course/

Marc Shotland
Director of Training
Abdul Latif Jameel Poverty Action Lab
Massachusetts Institute of Technology
shotland@mit.edu

Conference on evaluation for development May 2010

On May 20-21, 2010, a conference on evaluation for development will be held in The Netherlands: “Evaluation Revisited: Improving the Quality of Evaluative Practice by Embracing Complexity.”

This conference focuses on how evaluative practice can be improved, given the need to view much of development as a process of social transformation and, therefore, complex. Current evaluation practice has not yet embraced the full implications of assessing `the complex’ and existing approaches often fall woefully short. During the conference, participants can explore concrete evaluation practices that reconcile an understanding of complex societal change processes with quality standards, including rigorous, ethical concerns, appropriateness and feasibility.

Effective development is being understood in many quarters as revolving around people-centeredness and institutional transformation, thereby inherently complex, i.e. non-linear, emergent, unpredictable. However, discussions on measuring development effectiveness have zoomed in on methodologies suited to measure simpler, more linear interventions. The risk is that development, in order to facilitate its measurement, is reduced to simple interventions. These two mutually influential trends sit side-by-side in increasing discomfort for those who understand development as societal transformation.

The Conference aims to contribute to clarity in the development sector about what constitutes robustness, i.e. core values and quality standard, for evaluative practice in development that recognises the complexity of societal transformation.

You can find the flyer and registration form at: http://tinyurl.com/evaluation-revisited.

The Organising Committee:
Jan Brouwers (Context, international cooperation)
Irene Guijt (Learning by Design)
Cecile Kusters (Wageningen UR Centre for Development Innovation)

Social Audit Course – live online 4 two hour sessions.

Social Audit Course – live online 4 two hour sessions.

Social Audit is a way of planning and measuring organisational performance against social, environmental and commercial objectives. A Social Audit verifies an organisation in terms of its governance procedures, social and environmental responsibilities and its equitable structure, its internal and external stakeholder responses to operations, and then it accounts to stakeholders. It is particularly useful for Social Enterprises, NGOs, and not for profit organisations. The course is aimed at managers, staff and board members who are actively involved in organisational performance measurement and democratic planning.

Delivered by Freer Spreckley on behalf of Local Livelihoods at live@locallivelihoods.com

Info: http://www.uk.locallivelihoods.com/Moduls/WebSite/Page/Default.aspx?Pag_Id=168

Measuring the Progress of Societies

The Global Project on “Measuring the Progress of Societies” seeks to become the world wide reference point for those who wish to measure and assess the progress of their societies.

http://www.oecd.org/pages/0,3417,en_40033426_40033828_1_1_1_1_1,00.html

The Global Project is hosted by the OECD and run in collaboration with other international and regional Partners. A representative from each Partner organisation sits on the Global Project Board. The Board has two Co-Chairs: the Chief Statistician of the OECD (ex-officio) and another member of the Global Project Board, elected every two years. The two co-chairs are currently Mr. Enrico Giovannini (OECD) and Mr. Pedro Conceicao (UNDP). You can read more about the structure of the Global Project here.

Organisations can become involved with the Global Project in a variety of ways.  In addition to the Partners, the other affiliates are AssociatesSponsorsTechnical Advisors and Correspondents. The Global Project works closely with these organisations on planning events, training courses, a work programme, research projects and more.

Partners

Major International/supranational Organisations that play a key role in the overall Global Project, investing substantial resources – financial or in-kind – over several years and assuming responsibility for the management of the Global Project and/or for specific tasks.

WHAT we are doing

Mission statement

Global Project on Measuring the Progress of Societies exists to foster the development of sets of key economic, social and environmental indicators to provide a comprehensive picture of how the well-being of a society is evolving. It also seeks to encourage the use of indicator sets to inform and promote evidence-based decision-making, within and across the public, private and citizen sectors. The project is open to all sectors of society, building both on good practice and innovative research work.

Is life getting better? Are our societies making progress? Indeed, what does “progress” mean to the world’s citizens? There can be few questions of greater importance in today’s rapidly changing world.  And yet how many of us have the evidence to answer these questions?

The concept of progress (Latin: pro-gredi) was first used by ancient Greeks. And it is a concept that has exercised philosophers from many cultures ever since. Progress may refer to improvement. But to improve what? Since the enlightenment people have widely accepted that progress means an improvement in the overall well-being of humanity.  But for a good portion of the 20th century there was an implicit assumption that economic growth was synonymous with progress: an assumption that a growing Gross Domestic Product (GDP) meant life must be getting better.  But now the world recognizes that it isn’t quite as simple as that. Despite high levels of economic growth in many countries many experts believe we are no more satisfied with our life (or happier) than we were 50 years ago; that people trust one another – and their governments – less than they used to; and that increased income has come at the expense of increased insecurity, longer working hours and greater complexity in our lives. Much of the world is healthier and people live longer than they did just a few years ago, but environmental problems like climate change cast a shadow over an uncertain future.  Definitions of Progress
Indeed, it sometimes seems that for every action to demonstrate societal progress, an equal but opposite reaction demonstrates precisely the opposite.  And when the experts disagree, what hope do the citizens have to engage in democratic debate about their future and make the right choices at the ballot box? Access to accurate information is vital when we come to judge our politicians and hold them accountable. But access to a comprehensive and intelligible portrait of that most important of questions – whether or not life has got, and is likely to get, better – is lacking in many societies.
Concerns about this have been growing. And over the past 10 years or so there has been an explosion of interest in producing measures of societal progress. Measures that go beyond GDP to represent a broader view of the ways in which societies are progressing and regressing. Measures which are based on the values of a society, not those of a single political party or an elite few. Such sets of progress measures can help governments focus in a more joined up way on what really matters: they can foster a more informed debate on where a society actually is, where it wants to head, and – crucially – the choices it needs to make if it is to get there. By measuring progress we can foster progress.

The Global Project’s goals

The Project Mission Statement says that “the project is open to all sectors of society” and the Istanbul Declaration urges “statistical offices, public and private organisations, and academic experts to work alongside representatives of their communities to produce high-quality, facts-based information that can be used by all of society to form a shared view of societal well-being and its evolution over time”.

See more about “Who signed the Instanbul Declaration

Activities and outputs

Here is the list of our plans of action which we hope to implement over the next two years. Overview of Research Activities

• What to measure?

Encouraging discussions about the what?  To measure progress one needs to know what it looks like.  Progress undoubtedly means different things to different societies, and we will encourage and assist societies to have a dialogue about what progress means to them.

A good deal of the Global Project’s outputs involves discussions around the measurement of progress and its constituent parts.  Most people working in this area recognise that progress, and allied terms like well-being, quality of life, sustainable development, comprise various aspects of including economic, social and environmental factors.  But there is much less agreement on which aspects are vital to assessing progress, nor on how to label them.  This is quite natural: work has been undertaken independently, by different societies, with different cultures, histories, landscapes and issues of concerns. Indeed, there is no reason to believe that every society should share some common view of progress.
However, although different views of progress are to be expected, the differences can be a hindrance to those working in this field. Differences in terminology – what one person means when they talk about poverty can be quite different from what another person means – can lead to confusion that is an obstacle to fruitful discussion.  Such differences can also hinder research: someone wanting to investigate best practice in measuring “social cohesion,” say, might need to consult many different subject areas, depending on how “social cohesion” is defined.

The draft taxonomy or framework is an attempt to address these problems.  In particular, it seeks to:

  • Establish some common language for dimensions (aspects) of progress that are often labeled in different ways.
  • Sections of the website, for example, will be developed to describe best practices in measuring aspects of progress and the framework will follow this taxonomy’s structure.
  • The Knowledge Base – our online database containing hundreds of articles in the field – is organised according to the subjects of the papers.  We plan to use this taxonomy to define the knowledge base structure.

• How to measure?

Working with experts from around the world the Project will develop a better understanding of how progress can be measured – especially in emerging and complex areas not yet covered by statistical standards.  There is consensus that these areas (such as safety, human rights, different aspects of quality of life, etc.) are important but much less consensus about how progress in them should be understood and assessed.

Developing an accurate and representative set of progress measures for a society can be challenging, especially for developing countries. The Project will prepare a handbook and deliver training courses and other support for those who need it. If information on progress is to be used, it must be trusted and seen as accurate and objective: therefore, the Project will develop quality principles for a set of progress measures and will use them to judge whether or not to accredit a set of measures that wants to be associated with the Project.

• Ensuring that the measures are used

When good statistics exist they too often go unnoticed or misunderstood by a broad audience. New ICT tools have the potential to bring dramatic improvements: the Project will foster the development of new tools and approaches to help decision makers and citizens develop a better knowledge of their society using statistical information. Overview of Research Activities

Events

An OECD World Forum will be held very two or three years in co-operation with a selected host country.

The 3rd OECD World Forum on “Statistics, Knowledge and Policy” was held in Busan, Korea on 27-30 October 2009, read more: www.oecdworldforum2009.org

This OECD World Forum focused on Charting Progress, Building Visions, Improving Life and attracted high level participants with a mixture of politicians and policy makers, opinion leaders, Nobel laureates, statisticians, academics, journalists and representatives of civil society from many countries. The 3rd OECD World Forum, was organised by the OECD and the Government of Korea (Korean National Statistical Office).

The Forum focused on three major questions: What does progress mean for our societies?; What are the new paradigms to measure progress?; and How can there be better policies within these new paradigms to foster the progress of our societies? The economic crisis made these questions even more important.

The forum gathered close to 2000 participants from more than 100 countries including; politicians and policy makers, opinion leaders, Nobel laureates, statisticians, academics, journalists and representatives of civil society. Read more:  www.oecd.org/progress/korea

The Indian Statistics Office has confirmed its intention to host the 4th World Forum in India in 2012.

Future Events

Past Events

Budget support in Practice

aid harmonization, best practices, poverty reduction development partners, programme aid partners, africa, switzerland, mozambique, burkina faso, tanzania, benin, ghana, nicaragua latin america and caribbean, africa

A wealth of new reports and interviews on Budget Support in practice can be found on the website http://www.gersterconsulting.ch. The State Secretariat for Economic Affairs (SECO), Switzerland, asked Gerster Consulting to take a closer and practical look at budget support from various points of view. On that background a series of field reports and interviews were written to cover experiences made in Benin, Burkina Faso, Ghana, Mali, Mozambique, Nicaragua and Tanzania. They aim at illuminating chances and risks, opportunities and limitations of budget support in a practical manner with a focus on Switzerland’s experiences. They reflect the author’s personal opinion. The essays can be downloaded in three languages: English, français, deutsch.

http://www.gersterconsulting.ch/sites/res_budgetsupport.html <!–(Original link doesn't work? Use cached version)–>
Added by Richard Gerster to Aid Effectiveness on February 05, 2010