Male Speaker: Good afternoon and welcome to our Development Forum. Please take a moment to silence your blackberries and iPhones. Our program will begin shortly. Thank you.

Ladies and gentlemen please welcome to the stage the administrator of the USAID, Dr. Rajiv Shah accompanied by Susan Reichle and Dr. Esther Duflo. [applause]

Susan Reichle: Great. Well, wow! What an audience. Welcome everybody to today's Development Forum. This is really exciting and thank you so much for taking time out of your incredibly busy schedules to be with us here today especially our interagency colleagues who came over to the Ronald Regan building, thank you and our development partners who are joining us here today. This is really a great opportunity I think for all of us to really celebrate the opportunities to look at some of the development challenges with Dr. Esther Duflo.

Today's Development Forum really builds on the Development Forum that we had last fall. As many of you remember our conversation with our administrator, Rajiv Shah, who when we were discussing about the Presidential Policy Directives and we've had several Town Halls and different discussions since, but now with our first outside visitor and our Policy Seminar Series which we began earlier this year, a monthly opportunity really to hear from outside speakers about some of the issues that really challenge us in development. And that is really the purpose of the Development Forum and the Policy Seminar Series which PPL has begun, to give us a chance to step back from our daily pace, which is rather rapid, look at some of the critical issues that are challenging us today and to have an opportunity for exchange.

So while I will leave the introduction to our administrator, Dr. Shah to introduce our esteemed guest, Dr. Esther Duflo today, I really hope everyone takes advantage of this opportunity to really ask hard questions that we are all dealing with in various parts of the agency or the inter agency or as development partners. So we will have the introduction, the presentation and then go to a moderated panel.

For those who could not be here today, we are having this videotaped and we will have this on the website tomorrow but we will also have this on our policy pulse which many of you know comes out every two weeks.

So with that note, I would like to introduce Dr. Shah who needs no introduction but one thing you may not know of late is, [laughing] of late, and he is going to find out now, it's a Monday midday and everyone hopefully had a nice relaxing weekend, but he was on the road as he is always on the road meeting with lots of different interest groups and this interest group he was meeting with this weekend was actually up at Colby College in Maine, sort of the next leaders of the next generation and really talking to them as well as their parents, and I understand there was a Nobel Prize Laureate in the audience as well as Jack Conley one of our mission directors out in Georgia, his son was graduating and it just shows that he is constantly working on these issues whether up at Colby College or tomorrow giving a very important speech to the Chicago Council and then after that accompanying the Secretary of State for the 50th anniversary of the OECD, the Development Assistance Council celebrating the 50th and the importance of development. So we are so thankful to have the administrator here today with his incredibly tight schedule to join us for this interesting conversation so .

Dr. Rajiv Shah: Good. Well, thank you Susan and good afternoon everyone. I am really glad we all came up for this because this is very exciting and very important. I think Susan you will be picking up the pace on the Development Forums as we get more great speakers from around the world to come and share their ideas and insights with us so that we can take time out of our unbelievably busy and hectic schedules to just reflect on what is really the next knowledge frontier that should be informing the decisions we make and defining the work that we do going forward.

I first got to know Esther back in 2004 I think at the when we were part of a Joint Working Group On Impact Evaluations at the Center for Global Development and since then Esther's work and the work of the team at MIT and at Harvard that have used more I would say rigorous evaluation methodologies that I know Esther will talk about to define a series of insights and development have really started to change the way people think about what it means to be poor, what it means to live in the context and the environments that so many of us work in and focus on, and how people really use their own resourcefulness, their own sense of rational behavior, their own focus on survival to really make sure they are getting the best outcomes for themselves given the circumstances. And this work is so important that when we started and when I started here it was very important to me that we build a mechanism to begin to advance this kind of learning and allow USAID more directly fund programs and projects based on understanding what is coming out of the evaluations designed this way. And it was for that purpose that we asked Michael Kremer to join us from Harvard and I know many of you had a chance to work with him and that we launch the Development Innovation Ventures Program to really have a stepwise mechanism for saying we work on high risk, early learning projects and then as the data and evaluation builds we will scale it up through Phase Two and Phase Three larger grant awards that could be multicountry awards to really test the learning.

But I wanted share two or three thoughts on why I think this work is so important before asking Esther to talk us through on what is coming out in this book Poor Economics. You know, I don't know about you, but I have been somewhat disappointed and I was going to ask if we are taping this but I presume we are.

Susan Reichle: Yes, we are.

Dr. Rajiv Shah: I've been somewhat disappointed I'm going to get in trouble for this one, I've been somewhat disappointed by the debate that has gone on between people who espouse the arguments that Jeff Sachs and others have made about how to design and make development interventions and people who use various analytic methods to talk about the productivity of aid overall like Bill Easterly and more recently Dambisa Moyo. I just find I found the debate that is taking place about is aid good, does aid work of is aid bad, does aid not work to be a very high level discussion almost always defined by these gross generalities that for so many of you that are designing programs that you have seen things that work, you've seen things that don't work, you know how to put management focus and attention on things to get better outcomes and you know some of the risk factors in program and element design that can lead you in the other direction. It is never really captured that knowledge and that discipline of development. It has just been this high level discussion. And yet it has been the discussion that has defined the way so many Americans and so many elite thinkers around the world actually look at and judge aid. And so this book to me and the whole movement that Esther and her colleagues at MIT and around the world have really spearheaded is so important in rethinking how we make aid work, how we recognize the behavioral decision making of the people we are serving and design programs and projects around that. How we develop the discipline of improved evaluation methods and methodologies and make sure we learn from every investment we make in aid so that we can turn USAID's $23 or $24 billion of work into a large platform of learning and insight that moves the whole field forward. And it is not going to happen overnight. We need to create more space and time for evaluation, provide more technical support and resources to make that happen and really spend a lot of time thinking about how we build strong evaluation methodologies into programs and project design from the very beginning. And I hope that we will learn about all of those things as well as some of the very cool insights I would say that you have generated as we go through this.

I will just stop with one last comment and that is, you know, Michael Kremer had done I think a great study a while back on the use of chlorine tablets and when in the process of providing water you use them. And most people get them from kiosks and then use them for water purification and he, and Esther was probably a part of this, found that if you distribute that right at the point of water service delivery, you get much improved uptake and outcomes. And that was the insight that led us to do some things during the Haiti response that allowed for truckers from the Dominican Republic to distribute chlorine tablets with the water and lead to a 14% reduction in diarrheal disease in Port au Prince before when compared to before the earthquake.

So you all have the power in the programs that you design and in the decisions you make to put this kind of thinking to use and to do it in a way that will really dramatically affect outcomes.

So I am looking forward to Esther to your presentation. Esther as everyone knows, incredibly well accomplished economist. She has been I'm pretty much every 40 under 40 list there is. I am proud to be a friend and admirer of your work and we look forward to learning from you today.

Dr. Esther Duflo: Great. Thanks. Thank you very much. Thank you. Maybe the first lesson up here is I should have led Raj just present the entire book because he can probably do that substantially more eloquently than I will be able to do that. But since I am on the stage I guess I will give it a shot.

I want to start by I guess giving you a sense of why we decided to write this book. And which is also a bit of a sense of why our work is what it is. And it is the transition to it's quite a source but what one thing that does interest us a little bit is the fact that the development debate, the public development debate tends to be topped between this other extreme caricature and position. On the one hand you have sort of the success view that all we need to eradicate poverty is money, much more of it, but money. And in fact, poverty could have been eradicated by 2015 with increase in the foreign aid budgets. And on the other hand we have Easterly and Dambisa Moyo who claimed that in fact, aid is not the solution, in fact it's likely to be part of the problem partly because it creates a groups and constituencies which start fighting, it creates rents to fight over and people who have a stake in making sure that things stay the way they are.

The issue is that question whether aid can end poverty or not. It is a question which you can't really answer very easily. In a lot of cases what is being brought forward to answer this question are anecdotes and sometimes even the same anecdotes. Rwanda is a case that everybody likes because Rwanda post genocide made tremendous progress. It also got a tremendous amount of aid. So it could be a success of aid. On the other hand now President Kagame is saying he wants to wean his country off aid. So for Dambisa Moyo Rwanda is the success of the new aid movement. And, of course, how are you going to answer from this one example. And it is not only that one example is not enough even the experiences of all the countries in the world is not enough. Africa got tremendous amount of aid over the last few decades and the GDP per capita didn't really grow [inaudible] for the continent. However, you have no idea what would have happened without the aid. Maybe it would have been much worse or it might have even been better for that matter we just don't know. So that's not a question that we can predictably answer.

And it does not usually matter because at some level that's not the most important question. It's of course a very important question for us and I would say for you guys to know whether aid is important or not. But if you step back a few steps, the most important question is not whether aid works or whether money works but how to make money works because money is being spent not only out of aid budget, you know that very well, but out of the governments of all budget even in Africa where aid is an important part of the budget it is less than 6% of the budget of the country. So the vast majority of the amount of monies that are being spent on the poor of (inaudible) are on are spent by those developing countries themselves and the question therefore becomes how to ensure that the money that is being spent, either aid money or government money, is being spent in the most effective way possible.

And another reason why we find this debate not practically helpful is that because it asks a very big question can aid eradicate poverty it tends to stifle people's energy and enthusiasm. And here I give you a small experience. This is an experiment that was conducted by Paul Slovak (ph) [inaudible]. Slovak is a psychologist at the University of Oregon. They went to students sitting in the cafeteria in (inaudible) and showed them a picture of a girl, not this one but similar. (inaudible) and saying (inaudible) is a seven year old and she is facing the risk of dropping out of school and starvation and money can really help her. Will you give some money to Save the Children. And then they went to some other students sitting in the same cafeteria, randomly selected to make sure that they were completely comparable, and showed them statistics about millions people starving in Zambia and Angola. And then they asked them to give some money to Save the Children to help these people. And I guess you might not be entirely surprised by the results, but (inaudible) turns out to be a much better fundraiser. (Inaudible) raised $2.283 from the students and the millions the starving millions raised $1.16.

So then the researchers were interested in saying is it possible to erase this past. Is it possible to erase this tendency that we have to give more money to an identifiable victim then to identify a problem. So they went to two other groups of students and the first told them you know what, in previous experiments we've notice that when you show an identifiable person people are more generous then when we present the problem as a whole. And then they showed them the results. They showed them either a picture of (inaudible) or the starving millions and again asked them to give them money. So the question is do you think that that is sufficient to erase the past. Let's take a small vote. So erase the past to tell people in advance. Okay. No. Well I regret to say that you are wrong it does. But in a slightly counterproductive way which is people stop giving money to (inaudible). The starving millions don't raise anymore but no (inaudible) raise $1.36. Which I think illustrates one thing which is pretty fundamental about the nature human nature. Is our first instinct is to be generous and want to help. And then we think again and we think well this is such a small amount of money, it is not going to make a difference, it's just a drop in the bucket and a bucket probably leaks anyway. [laughing]

So I don't really need to do anything and so we kind of hold back on our instincts. And a lot of the discussion about aid are kind of this bundle of movement between one and the other.

What we want to do with this book and what we want to do with our work is not only make the amount of money aid more effective, but also show people both in developing countries and in our country that the problem is not even though it's enormous, it can be broken down in a set of issues that are identifiable, that we can think about, that we can see results. We can think about education, how to get kids into school. We can think about making the schools better, we can think about immunizing the children, we can think building roads, etc. and each of these problems has a set of concrete issues that need to be understood and solved and we can do it step by step. And that is part of the whole philosophy of the enterprise and the whole philosophy of the book and we hope that in this way we can slightly change the mindset.

So instead of walking you through kind of a million arguments I thought what I would do in the short amount of time that I have is just to take three examples of things that surprised us along the way and that might surprise you too and once you find out when you forget about the large question, go to the small question and try to understand a specific problem.

So the first one is concerns education and it says tracking is good for everyone. What is tracking is to separate kids in groups of different achievements. Education is in a sense a story of success. Over the last few decades many more children are going to school. In a country like India for example pretty much every school of primary school age is in school. But one problem is they are not learning all that much. In 2005, 40% of children in grades 2 5 could read a paragraph, 30% of them could read - could do a very simple division, all of which should be at grade one competencies. In 2005, the survey was done again in 2006, 2007, 2008, 2009, 2010 and the results have not changed. And this is despite a huge influx of students in primary schools. And this is not unique to India. This is a problem everywhere. Students are now in school, parents have bought in to the education agenda and they are not learning very much.

So when you have this issue now it becomes, well how do you make our school better? How do we make sure kids are learning something and you could think of a variety of ways you could do that. You could provide [inaudible] schools, you could hire extra teachers, you could give -- you could build latrines, you could distribute sanitary pads etc. etc. So there are number of things you could do and then there is this one -- so the question becomes well what would be the most effective way to improve the quality of education. One possibility is perhaps not a very revolutionary sort it is how much teaching kids at the level where they can learn. So that may sound like rather mundane but a lot of schools look a little bit like that. This is a very small school in Monaco but what is hiding in this picture is that the kids are so different. There are few children who clearly should not be in school whatsoever who are way too small and then there are these older kids etc. so you have kids that have come to school with very [inaudible] backgrounds, different age, some of their parents can [inaudible] they have gone to preschool or they have not gone to preschool and everybody is brought in to a classroom. This is a small one, but in Kenya it would be a classroom with hundred students in grade one and then they're given the same curriculum. In the past that is probably adequate for the best students in the class. And the result is that students get very lost very quickly. So one thing that Michael Kremer [inaudible] how about we try and separate the classes so that kids are more homogenous. And so we conducted a randomized experiment in Kenya and the experiment is the following, they give some money to us to give to the schools so that they could hire a student teacher for grade one. So the classes were cut from 80 students to a class of about 40 kids. They are not so interested in the effect of dividing the classes although that is also interesting, but we were interested in the effect of if we compare schools that will divide to equal classes versus two classes, tracked by prior achievements so we took the grades from grade one and the schools divided the students by prior achievement. And so for the students who are at the top of the distribution, the students who are doing very well this would be all good right. They are in between themselves and the teachers can devote them more attention so they should all benefit. Do you think this would be good for the low achievement kids or bad for the low achieving kids? Let's take a good vote. Why of course I had to respond to it already. So it is good. It actually happens to be that students and the test scores after 18 months in the tracking and no tracking schools for the students who are initially high achieving and students who were initially low achieving the schools were randomized so there is nothing different about the tracking and non tracking schools and you can see that both in the high achievement kids and in the low achievement kids students are doing better if they are in the tracking school than in the non tracking school.

So this is something that might be surprising because you would think well for the low achievement kids they are losing their most talented classmates so that is a loss and therefore, there is a field that tracking would actually increase the inequality between students by helping the high achievement kids but hurting the low achievement kids and the reason why it did not hurt in this case I think is because reflect, something which is quite fundamental about the education system in Kenya and in many other developing countries which is - it's -- the education systems which have a tendency to be ill at best. These are two reasons. For historical reasons because the education system have been passed down from colonial administration times and colonial administration's objective was to form a very small elite and also because of the current expectation and demand by parents. So a lot of parents, education is a ticket, a lottery ticket I should say for a government job. But it is only a useful lottery ticket if you can get at least a grammar school education so parents tend to believe when you ask them that there is no benefit in the first few years of education whatsoever and the benefits come up later. Hence they are only interested in their kids going to school a lot if they can do very well. Therefore, even parents of this elite best and private schools are no better and they are also pushing students. That means that the curriculum is very demanding, teachers teach to the curriculum. If students get lost very early on they get lost forever. So in a sense fixing the quality of education might not be all that difficult because what it might require, while maybe it is difficult but what it required is less something very sophisticated in terms of changing techniques and things like that, but really changing the forecast and saying well the goal of education is to teach everybody to read and count. This insight I have been finding Kenya very similar results were found in India and now being exported. This is a picture from Ghana where the government is trying a very lasting experiment to which has been on exactly those insights. Which gives you an idea of how from one experiment you can understand the problem and from your understanding of this problem move on to an influence position. If the experiment works in Ghana then they'll adopt this as a policy.

Okay, here is a second example. Television is more important than food. That is not an advertisement for something. It is something which was actually quite surprising to me as I started doing research on developing economics. I was 13 during the Ethiopian Famine and the Live Aid concert so my view of what a poor person is and how poor person live has been formed very deeply by the pictures of Ethiopian children with distended bellies and this is really when I can be remember thinking that there is something that I need to do about this. So for me, being poor for a very long time and being hungry. And this is not I am not learning this view. For a lot of people being poor means being hungry. In fact, in a lot of countries the poverty line is the budget that would allow people to get some minimal acceptable debt in the country. This is not just a view; this is a view that informs policy in a very important way. So for us a very significant amount of aid goes in the form of food aid but, also countries spend a lot of money trying to -- subsidizing food [inaudible] stubs and a lot of energy lugging food grain around which is an extremely inefficient thing to do. In India for example, about 20% of the food grains get lost somewhere along the way. And the reason why they are doing that is because of the very strong view that there is a nutrition base poverty top. And a nutrition base poverty top works in this way, you are not eating enough therefore, you are not very productive therefore you don't make much money, therefore you don't eat enough therefore you stay poor. One wrinkle with the nutrition based poverty top is there is not much of a sign that in a lot of countries that is the real issue. Because if it were a real issue then we would see that as soon as people become a little richer they should want to spend all of this extra money on food. India is a place where the poor are becoming richer and it is also a place where people are eating less and less. This is since -- between 1983 and 2004 the fraction of people who are eating less than 24 calories a day and has gone up not down. So as people become richer they eat less and less. On the other hand they don't report being more hungry as people become richer they complain less and less about hunger. Is it that Indians eat enough? No. 50% of the adult women are anemic. 50% of children are stunted. 25 are severely stunted so there is a nutrition problem but it is not perceived as stuff. It's not a hunger problem; it's a nutrition problem which needs to be addressed quite differently.

Here is another example. Very interesting experiment and another [inaudible] experiment from China which mimics the idea of making grain cheaper. So it's an experiment by Rob Jensen and Aaron Miller. What that it is that distributed vouchers to make rice cheaper in a place where rice is a staple. And so when rice price goes down by 10%, by how much do you think rice consumption goes up? Zero, 10? [off mic comment] It goes down. I had a minus there. When rice price goes up but go down by 10% rice consumption drop by 2.3%, calories consumed go down and the quality of nutrition does not go down. So what is going on here? Well people are feeling a bit richer because rice is such an important part of their diet and they are now get it for cheaper and with this extra money they do other things. They buy shrimps or other things. What does this mean? What does this mean? It does not mean we should stop worried about food but it means we should stop worrying about it in a very different way. We should start thinking how we should start taking seriously the preferences of people. We should start realizing that like you and me they are not maximizing physical fitness. They are maximizing happiness which involves other things than calories. It involves taste, it involves cell phone minutes, it involves other things like that. And we should say given that, how do we have a nutrition policy which is a nutrition policy and I think what is very important is to make sure that it is easy for people to get the right nutrient. So you could think of [inaudible] iodine, you could think of a new variety of costs that has better nutrition capabilities in there.

Here is my last one. The proof are from risk but they do not want to buy insurance. We talk a lot about the private sector of having to play a very important role in development and insurance is a key factor. Of course, the poor are subject to all sorts of risk, the weather, the health and things like that. They also run their own business and risk is costly not only once it hits but even in advance because people are becoming extremely conservative when they feel that they could be risk so these are people in Kenya who go traditionally, they don't go hybrid means because they are worried about the extra risk that is put on them. That means that makes them poor just the fear of being attacked. So given that you could think where are the insurance -- where are the insurance company? Insurance could be the next billion dollar market of opportunity since people really would value the reduction of the risk and the problem is that people are not willing to buy the insurance. This is a weather insurance product; this is a randomized experience that was informed in Ghana where people were offered insurance at value prices. The insurance is a very simple [inaudible] product which gives you something if the weather -- the rainfall and the weather station goes down a certain level. If it is free, everybody wants it but if it becomes to become 9.5 CD which is what you need to charge to cover the risk only 30% wanted and at 14 CD which is what the insurance company needs to charge to make money, no one wants it. There are reasons for that. I think the main reason is this type of product that insurance company can offer are full of holes because if in fact, it doesn't rain -- it has rained a lot, but not the usual you don't get paid. So people find it very insufficient.

The lesson I want to take from it is that it is not enough to say oh, that is just a missing market. It is a market that is missing for some very deep reason. It means that perhaps the insurance will need to continue to be subsidized or it means perhaps that we could invent better product but certainly that the solution is not to say oh, the market will just go and take care of it. There is a need for better design.

So where does that take us? Where does it matter? I think it matters because the reason why a lot of places don't work so well in the fight against poverty is not because of some really insurmountable value that there is between us and fixing the problem. Usually the problem is what we call the tri I problem which is ideology, ignorance, and inertia. We saw an example of that and just one point example of that in India where a village education committee. So the village education committee are involved in the probably reasonable idea that people should be involved in the services that help them. So every village in India has to have a village education committee. So in [inaudible] we went to those villages and we try and find out how they were working. We found that 8% of the people knew that the committee existed, 2% knew what the committee was supposed to do, and first I think it was this one which is a quarter of the committee members had no idea that they were committee members. [laughing]

So this is a policy that was conceived in ideology right in complete ignorance of the reality in the field and people were very surprised when we showed this results and consists because someone is checking the right boxes. What does this mean? Is this a pessimistic message or an optimistic message? In my view it's a very optimistic message. It's a message that says look, in a lot of cases we can fix things and in a lot of cases things are wrong not because of some deep reason but because no one took the trouble to understand what the issue really was and tried to fix it. That means that we can go, that usually change in the matter of understanding what the problem means and leading the charge. So that is for me a message of deep optimism, but also responsibility because if that is the case it means that there is no hiding behind the problem is so difficult, there is no hiding behind this is all about corruption, there is no hiding behind this is really the matter of the government, this is really I am just your cousin of the big machine. Every one of you and it is also true for every one of the government people who work with the NGO people, people in businesses have like in small role to play and how to make one thing work better. Part of this is about thinking about issues, part of this is about keeping eyes open about facts and accepting facts when they don't correspond with what you have to say. A lot of this is about [inaudible] and excepting fear and coming back and trying something new and I am hoping that our work is sort of kind of a first step of leading us in that direction. Thank you very much. [applause]

Susan Reichle: Great. Well thank you so much so much Esther. That was just a magnificent presentation and just hearing the reaction from the audience I know there are lots of questions out there. But as the chair or the moderator I just would like to ask you first and we did not coordinate this which was so interesting because I did not know beyond her writings really what she was going to present today but one of the things that really struck me as I was reading about her writings and her work and with her colleagues is that it really seemed to move us beyond the Easterly-Sachs debate about whether aid works or not as the administrator mentioned. And she clearly highlighted in her presentation. And actually about how it works and what are -- what is the evidence about what works and what does not work and how do you move that really to a grander scale as we are all working I think very diligently on lots of different activities at different levels from the small activities, working with local partners to the larger working with our international partners. And how do you learn from those interventions in order to scale them up? And obviously as she was talking about the example here today about the price of rice in that going down and clearly the changes in attitude and I just want to take the opportunity to ask you about as you gather this evidence and as you then try and push it up which is what I think our administrator has really asked us to do and the president has asked us to do through the presidential policy directive of really creating policy, what are some of the lessons you have learned about scaling up and taking sort of those small innovations I read about what you did early in your career in India and a very small intervention that really resulted in national policy change and what recommendations would you have for us here at USAID and then for the administrator really to respond as he is working with us and how to move this forward.

Dr. Esther Duflo: So what I would say is you are in some sense ideally placed as you are deciding to do this because when -- if arrive and say well there was this we have all of this evidence usually do you don't try to convince people to adopt a policy based on just one experiment, but things kind of accumulate and for example, on education tools, such a large number of experiments we get really convinced of the importance of primary education and focusing on the basics so kind of ready to move and say this is something that should be incorporated in national policy but I can go and say that or one of us can go and say that in our -- on the basis of our experience, but you at USAID and as such have both a traditional of technical expertise and an amount of credibility, institutional credibility that we don't have and therefore, what you say when you go to the field carries a weight that what I say don't carry doesn't carry. And hence you could think of yourself both as producer of new evidence by evolving the project that you are supporting but, also as consumer and consolidator intermediary in order to try and make sure that the policies that you push either with your own money or as technical advisers are ones that are based on something that we know. That is collectively known.

Dr. Rajiv Shah: I would just build on that. I think to Esther's point as a producer of evidence, every single project we do, every program whether it's service delivery, infrastructure, anything else is a potential opportunity to produce evidence about what works and what does not work in development. And I think what we have learned from Esther's work and obviously it has worked in so many others is that in order to seize that opportunity, we have to bake the methodology right into the study design. And that means thinking from the beginning of a project about what are you going to compare this to in order to make a convincing case that this had impact or this did not have impact. I will give you one great example. Our team in southern Sudan has been doing a project using radios to help improve the quality of education in schools for kids, you know, who are I met the students in what they called a second standard, but they went all the way up to 13 years old. These were not six year olds. And they found through using a very rigorous methodology that they got a significant percentage improvement in learning achievement because of the supplementary intervention of having that radio there doing standardized curriculum via the airwaves. That is so important because had they just said we do the radios and in the schools here where we are using them we are getting the following results and done the process work of counting those results, that would not actually tell you does the most costly part of this intervention, that radio program and the physical solar powered radio actually lead to a statistically significant impact on learning attainment. But they had done that. And so as I was walking through the project there was a whole wall with probably nine or 10, one or two pagers on success stories and all of which were interesting to read, but not particularly compelling because they were stories of the projects we did. And then there was one bar chart like the one Esther put up on this radio program that had been evaluated rigorously and I took that and kept that with me because it said this is compelling evidence that I can then use to say we should be doing more of this, we should be investing in southern Sudan. We can -- we know how to use aid in a way that generates results so if there is one point I would leave this group with it is to think about what it takes to produce that kind of rigorous evidence and really design that into your programs from the beginning.

Susan Reichle: Great. Thank you. Before throwing it open to the floor one of the things we like to do the development form is actually hear from our Foreign Service Nationals which really are the core of our agency oversees. And when they heard you were going to be here today we received lots of questions, but one I just want to highlight from our Program Specialists of Economic Growth in Sri Lanka, Rebecca Banboondala [ph] asked why do development assistance grants in your opinion create an economic dependency mentality among the poor instead of economic empowerment? Is this because of poor packaging of donor funding or is this the result of development agencies rush for quick results during a short lifespan and I guess I would want to add a little bit too that about the quick results. One of the things as you're obviously known for your randomized trials which we have really adopted in our evaluation policy and looking for every opportunity to do that but one of the concerns we hear from our staff is there -- it is incredibly expensive, it is long term and how do we address those issues as we are really trying to dig deep into the evidence of what works and what does not work in a very challenging budget environment obviously that we have as a US government agency.

Dr. Esther Duflo: I guess I would rephrase the question slightly. The question is why and I would say whether. I don't know if it -- I don't know if it is a fact actually. In fact, I think it is more of a law than a fact and the one experience I know that tried to look at this question found the opposite. In the days of bed nets -- a big [inaudible] debate is about bed nets and whether they should be given away for free or people should pay a little bit for them and one argument for why people should pay a little bit for them is because they are not going to use them if well if they have not paid for them and another argument is the dependency argument which is people who got the first one for free would not want to pay anything in the future number one. And number two, if I got one for free, the administrator is going to wait till it comes for free for him and is not going to buy one. So that is something which you must have heard over and over and over again. So Pascaline Dupas did a simple experiment where she distributed vouchers for reduction in the price of bed nets from zero to different prices like she had a bunch of prices. And what she found his number one, people who get bed nets for free are just as likely to use them properly as people who had to pay for them. Number two, people who got the first of bed nets for free, six months later she came back and offered everyone the opportunity to buy a second bed net at half price. And the people who got the first one for free were actually more likely to get the second one at half price then the people who had not gotten the first one because it was paid. The reason is people did not get used to handouts. They got used to nets. And they realized that the nets actually are helpful and it's nice to sleep under and less malaria and so they were more likely to get a second net. The third result was that if I got a net for free and he didn't, he had to buy it, he was actually more likely to buy it because once again he learned from me so I think from this study we are not learning that people will never get used to aid, but we are learning that with the kind of there is a little bit of a tendency to use that as a cliché or as a caricature and we should kind of question these things all the time if indeed they are true, when they are true and this needs to be dug in the design of the program to avoid it or if they are not then we are not worried about it and we are distributing a large number of bed nets.

Susan Reichle: Thank you. So the floor is open for those who would like to ask questions.

Jonathan Sheppard: Thank you for taking the time to be with us here today. Jonathan Sheppard, USAID's Office of Learning Evaluation and Research. We have heard a lot today about relatively I would say the set of interventions that you discussed are somewhat I would described them as discrete and it is relatively clear that we have a lot of work to do as far as asking those tough questions about relative cost effectiveness of various approaches to achieving certain development objective. It is less clear that we have any clarity on where to go for our colleagues over in the field who are trying to answer what I would describe as more complex development questions like how to contribute to effective governance structures in Afghanistan or addressing land tenure issues in Zambia that maybe lack a clear opportunity for a counterfactual. So what would you say to those who would raise this point?

Dr. Esther Duflo: So let's leave Afghanistan aside. I think I have very little to say about Afghanistan. That's -- I think that is I am just so much not an expert that that would be terrible. But -- [laughing]

That is a question which -- some version of this questions we hear a lot and we are asking ourselves a lot which is well, that is fine, you can kind of try and tinker around with things and make things a little more effective, but at the end of the day, governance, politics is going to take over and it is really politics which is prime and institutions rather than policies. And we have two things -- or I guess three things to say about that. The first thing is that is true there are policy constraints and the second thing however, is that there is a tremendous amount of slack within those policy constraints to take a relatively well functioning democracy like India and they have really significant progress, there is a lot of progress to make within that. And even in not so good countries you can have [inaudible] so you take South Sudan a lot of political problems and maybe starting a new war was enough. And yet the question of whether or not they can improve the quality of education is pertinent because in the context you can make improvement. The third one that is probably the most direct answer for your question is that it is actually quite productive to think about politics a little bit in the same way as you think about fixing schools which is at the end of the day politics is not some magic thing where you put people together and at the end of it a policy emerge. Politics is a set of rules, a set of quite detailed institution who can vote without a vote with a paper ballot or electronic voting if there is decent transition, who is invited to the meeting and how and all of the procurement holds, how many people need to sign a form, all of this things are huge influences on outcome and you can experiment with those very easily and actually there is a lot of studies now on politics trying to incorporate political processes by providing more information to voters which has some very, very encouraging results about how you make politics work better and so that is kind of one way to answer your question.

Dr. Rajiv Shah: Could I just say on Afghanistan if you look at different sectors in the portfolio there and I'm sure we have folks from our [inaudible] team here. They are spending a lot of time right now figuring out how do you maintain or improve the level of outcomes while dramatically reducing that cost structure of achieving those outcomes. And I think their insights in this book and in the body of work that Esther both represents and has supported that offer a huge, huge insights. When to sell farmers fertilizer, how to sell it instead of giving it away, how you make the health system more sustainable, how you implement and use mechanisms like mobile payment systems to reduce the leaky bucket problem in building a civil service. And those are not small technical fixes. It's a collection of getting those things right that will determine success or failure on the governance and development side over the next decade. And I just want to congratulate the [inaudible] team and some of their supporters here for taking that seriously, doing their auditing, all of the programs to ask those kinds of questions. And I think that all of these insights apply in that setting as well.

Susan Reichle: Great. So another question.

Carl Hoffman: Good afternoon. Carl Hoffman from Population Services International. Implementing partner and a big distributor of bed nets. And thanks very much for this fascinating conversation. I look forward to buying the book Dr. Duflo. A comment and a question. The comment I suppose is the three I's that you mentioned, ideology, ignorance, and inertia. It is surprising to me how much that applies in Washington too [laughing]

So it is not just in India. The question is though I suppose related to that. Your charge to us to learn and partly to learn by failing fast is a really powerful idea, but in the current climate and in with the political climate that we all face and certainly the administrator faces, that the institution faces, that is hard. Perhaps it's really a question for Dr. Shaw. How do we go about failing fast and learning in the environment that you are trying to lead this institution in?

Dr. Rajiv Shah: I would ask Esther to start and then I'm happy to answer that [laughing]

Or do you want me to start? I'll start. Okay.

Dr. Esther Duflo I say something. One is everybody knows that everybody knows that that is the system. So I think no one to the extent that things are not evaluated and everything is a great success, two page stories of success that you are talking about, no one is Congress knows that, everyone knows that, people who are paying or giving on their own taxes they know that too so there is a general climate of being just extremely suspicious about anything that an institution like you claims in terms of success because it is not backed by the occasional failure that indeed we are kind of rigorous about things. So I -- maybe it is wishful thinking and I would be very keen to get your comments on that but I think that if the institution was willing, and I think it is willing to go all out and say well we are going to be very rigorous about evaluating at least some things and for these things we are going to tell you this is what we found, this is how it works, this is how it could work better, that is the next step. And on this [inaudible] portfolio which does not have to be all that a giant part but that is part of the portfolio, the lessons learned here we are going to do use to inform the entire rest of the portfolio and therefore, you can rest assured that what we are asking you to spend 80% of the budget on or 90 is supported by this 10% of the budget of 20 that we are being completely brutally honest about whether it worked or whether it doesn't work. I don't live in Washington. I look at this with a little bit of dismay, but [laughing]

It seems to me if someone said that to me then I would say that that is kind of the way to go. That is why, that's how drugs are being tested and that is why we trust what the drugs we buy is because we know that money has been spent on trying things out, a lot of public money has been spent on drugs that fail for good reason. That is the only way to make progress.

Dr. Rajiv Shah: I would just endorse what Esther said completely and I would go a little farther and say that I think we should actually move away from even using terms like success and failure. I don't think in this field anybody expects that we are going to turn a switch and change the landscape of a country overnight. And I don't think anybody has any illusions as to the fact that when you spend resources in these environments you are taking risks and doing so. And I think if we could move our language and our approach and our mindset completely to from success and failure to what works and what does not work, that would be a huge shift, you know, because no one is really buying the success story pages. You know, but they do buy a rigorously described example of why something works. And generally we have the data and there are methodologies we can employ to say this works and here is how we know it. Even in Afghanistan we use the DSF, the District Stability Framework tool and we can say these projects even in kinetic environments worked or did not work at changing the set of outcomes and I think that is basically where we have to go. The other thing I would suggest for our PPL team perhaps is as we do more of the policy task teams and the papers related to that as we did an outstanding job David Barth and his team on the education project, including in those policy reports the studies and the references to the knowledge creation that we relied on to lead to the policies we are recommending that adds a lot of value because it helps our staff around the world and it helps our partners around the world see why we are making certain decisions so I would just move away from success-failure and think more rigorously about what works and what does not work.

Susan Reichle: Great, I think a lot of people out here would applaud that because we been often been forced to look things as success and failures and so we only have a couple minutes remaining are there any questions out here? Leo?

Question: Thank you Dr. Duflo. I am in the Office of Policy and one of my jobs is to try and figure out at what point is there a critical mass of evidence out there that would then justify us building a policy on? And one of the I think strongest criticisms of the kind of work that your shop is doing is that people who don't like the answers will say well, we know that in a very specific context, in a very specific environment, but surely that is too narrow a base to draw upon for a large policy for an institution and one of the things we do for good or ill I think is to be able to say we are going to do this sort of policy in a very large number of places we can't have -- we are trying to be customized but there are limits to how much you can customize an experiment in an institution this big. And so I guess the question is to what degree at what point do you feel comfortable there's enough of a critical mass of research to be able to comfortably advise that this is the way to go? Thanks.

Dr. Esther Duflo: Thank you. That is a great question. So I think there are two ways to answer, there are two answers to the question, not two identity ways, one is the indeed the replicating projects in different context usually they are replicated with some amount of changes, but to adapt to the concept is very much the same idea and that, that progressively gives you some confidence that the results generalize the cost different context. And the second answer is sort of what we are trying to do in this book which is to say well all of these trees if you think of them as individual studies not necessarily experiments actually but also different kind of studies, how do they kind of adapt to a forest that an understanding of the structure of the problem. So that is when I was talking earlier that's what I try to do with education which is to say this tracking experiment it is not just one tracking experiment in Kenya it is actually six with a number of results of very actually different projects that all point in the same direction of saying the structure of the problem is a curriculum that moves much, much too fast for most of the student and therefore, an education policy first priority would be to kind of reverse that. And so you think of each project the first ever experiment was Michael Kremer textbook experiment which found no set of textbook which maybe was a bit puzzling at first but fits very nicely into this because the textbooks are in English and most students can't read English. Of course, why are they in English? Well that's because the curriculum is in English so that textbook should be in English. So that's -- this is a very different experiment then tracking, but it fits into this general knowledge of what is going on. And so for example, for education at this point very confident to say any policy that reorients the teaching towards something that is more modular and that is more focused on core competencies and also incentives of teachers in this direction would help. So that is kind of there has to be a little bit of an inductive-deductive back and forth between the experiment that are accumulating and some things to tie them together in a bundle which usually can give you ideas of what to do next and therefore, may be what is the next experiment. And I think that is what happening now in Ghana.

Susan Reichle: Well thank you. Unfortunately we are out of time. I think one of the lessons learned is we always need more time, but I want to thank you very much for all of that you've gave today and offer the administrator to say a couple of words. [applause]

Dr. Rajiv Shah: Thank you -- thank you very much for coming. I think based on the great turnout here Susan we should do more of these more often.

Susan Reichle: We should. Absolutely.

Dr. Rajiv Shah: So good thank you very much. [applause]