| 0:36 | Intro. [Recording date: July 6, 2011.] You are critical of what might be called two extreme views of development and aid. One is associated with Jeffrey Sachs, who argues for large top-down approaches--from the outside--and one approach is associated with William Easterly, who argues that most aid is wasted and only a bottom-up, internal approach will work. What do you see as being incorrect with these two views? Not so much that I quarrel with the top down as much as that we know the answer to any problem and the answer is money. I think more often than not the problem isn't to be solved by just putting money in the most conventional way onto the problem. You have to think of why the problem is the way it is, try to understand it, try different solutions out, do the kind of experimental work we do at J-PAL--tryouts, strategies. And then you get to the answer. But that takes a while and it takes a lot of care. It's really not money or no money. It's money, but money with intelligence. You argue that the most effective, maybe the only effective way to get that intelligence is through a recent approach that has been taken, sometimes called RCT--randomized control trials. In layman's terms, running experiments with two different groups to try to measure the impact of a particular kind of intervention or form of aid. Talk about how those trials or experiments are structured and give us a flavor of how they are executed on the ground. Let me start by making a caveat. Our book is very careful to not quite say what you said. I think we are very careful to say that it is not just a matter of randomized control trials. It's a matter of presenting maybe the handy tools to answer these questions. But the book makes the point that it's much more fundamentally about knowing what the problem is and understanding why what you are doing is going to solve the problem and trying out different solutions. Sometimes the easiest and most effective way to try out the solutions and know the answers to those questions is what is called a randomized control trial, but that doesn't mean that it is by any means the only or even sometimes the most [?] of knowledge. Sometimes you can learn from many other facts that are staring at you. I think the main point we were making is broader, which is that it's very important to kind of understand that the on-the-ground mechanics of particular interventions. What is it that makes the particular problem we are trying to solve a problem? Why do we think that the intervention we are about to undertake is going to solve it? There are often glib assumptions about the causal sequence of the interventions to the solution, which we are liable to question and often find are inadequately thought through, the way policy is done. Establish exactly what we are trying to say. Why so much emphasis on these randomized control trials? What's nice about randomized control trials--two really nice things. First, they make it relatively easy to get a reliable answer to the question: Is this particular intervention working the way we are claiming it is working? The reason it makes it easy is it solves the perennial problem of all such evaluations, which is how do we know what would have happened if we hadn't done an intervention. Suppose you wanted to know the effect of providing computers in school. You choose some schools to have computers, and others not to have computers. Or the government so chooses. And then you find that at the schools that got computers, actually the students learned a lot more. Now, the challenge is to figure out: Does this mean it's because of the computers that they learned more, or is it because the government gave the computers to the schools where the students were more enthusiastic and were more interested, and that's why the kids learned more. You can easily conflate the reasons why the government gave the computers to those schools in the first place with the effect of the computer. This is a fundamental problem of all attempts to evaluate. This is the same problem when you look at the effects of microcredit. You look at the women who got microcredit and they are richer than the women who didn't get microcredit. But you don't know whether the microcredit is something people get on purpose--they go and find out about microcredit, join the group, get microcredit. And there's no reason to assume that the people who take the trouble to get microcredit are like the people who don't take the trouble to get microcredit, at least at this point of time in their life. Some of these people are enthusiastic about doing something. What randomized control does is it solves that problem of inference. It basically says: this school and that other school, or these hundred schools--their names are put into a hat and we draw out 50 of their names at random. And so the schools that did get the computers or the women who did get the microcredit are chosen at random, and that gives you the advantage that you can compare the two groups. There's no difference between them; decide by lottery which will be in each group. And therefore you can compare them without worrying about it. That's one big reason to do it. There's another reason, less emphasized but equally important. Suppose I wanted to know the effect of a particular computer program on learning. How would I go about finding that out? If I look in the world I might see that computer program used in some school. But it's not the case that it's just that program that's the only difference between those schools and other schools. Those schools might have different textbooks or different kinds of teachers. The blackboards might be different. You don't see one difference between schools; you see ten differences. Even if you just wanted to compare schools, and some schools have computers, you also get along with that the fact that those schools have many other different things. Very hard to unpack the effect of one particular thing, of a set of interventions into the constituent effects of individual interventions. One nice thing about doing an experiment is I literally choose: these 50 schools will be picked out a hat and I just give them this one extra thing. So, I can figure out what this one extra thing does. That way I can really, step by step, figure out what are the causal channels through which I can influence schooling. |
| 10:05 | Let me ask you a question about that. It's true that if you do it randomly, effectively as in a lottery, as if you are drawing names out of a hat, you have not biased the experiment yourself. But it does not follow that the 50 schools that get the computer program and the 50 schools that don't are effectively the same. They can have differences. So there is still going to be some statistical measurement of the program, I assume, to tease out the fact. You are exactly right. What a lottery guarantees is that these two groups are statistically identical. They are drawn from the same distribution--a word statisticians sometimes use for this--so they are identical in the same sense as if I toss a coin 50 times, and then I toss a coin another 50 times, those two groups of 50 tosses are identical. There is no systematic difference between them; and we can calculate exactly how much difference between them could arise out of random chance. So, it's not that we assume they are identical; but because we know how big these groups are and we know the laws of chance, we can figure out how much of the difference between these two groups can come out of pure random chance. Which is the point you are making. Therefore we can say that this difference is so big that it could not have come out of random chance, given the way we have drawn these things out of a hat. You are right--there is a role for statistics, and that role is to tell us how much of this difference could have come out of random chance. But we know exactly how to do that. Advantage of doing it is that we have a handle on the nature of that difference. Let me try to clarify that; I'm a little confused. If I try a particular software, let's say a learning tutorial. I make it available to students in 50 schools. And there are another 50 schools that don't get the tutorial. And then I try to measure after the fact its effectiveness. So, during the period when I decide which 50 schools, that's done randomly. But it could turn out that when I choose the 50 schools randomly, it still could be the case that the backgrounds, the socioeconomic backgrounds of the 50 schools are not literally identical to the 50 who don't get the tutorials. So, I would have to control for that, say parents' income or educational experience of the students that they might have that's different across the two samples. Don't I still have to control for those factors independently, to measure the independent effect of the tutorial? No. Actually, you don't. In fact, if you did that then some purists would say you actually vitiate the experiment. You are relying here on what is called the law of large numbers, which is: If I have 50 schools drawn out of a population of a hundred, then the probability that the 50 is systematically different from another 50, the probability that the first 50 is different from the average is something we can compute. What we are going to do is say that if the observed difference is bigger than the difference that could have happened out of random chance--we are going to let the data tell us the answer to that question, other than sort of trying to fix the random differences. One way to think about is imagine we had a million schools, and we drew half a million schools. By the law of large numbers, by the time you have a million all random differences will have canceled out. We know the probability exactly of how much--the probability that the mean performance of a group of half a million will be different from another group of half a million randomly drawn from the same population--we know exactly how unlikely that is. So, we can say with probability .99 these groups have exactly the same distribution of attributes, or probably .999 I know exactly how close these distributions are. I know how to control for the random differences between these groups rather than trying to actively control for it by putting in measures of those things. That would be a different way of doing things. I'm with you on a million. On a hundred, or 60, I understand it's still pretty high. But the bottom line is the methodology of how these are measured, you do not try to control for any differences that may have arisen. You assume they are the same population. I think you are saying something that I am slightly disagreeing with. We don't assume they are the same population. We assume we have randomized correctly and adjust our inference based on the fact that if I have drawn 50 out of 100, then the probability that they will [?]. When I report a result, any empirical study, I say with 95% probability the effect of this is greater than zero, or greater than 2 or greater than 5. That the claim that is always made. So, when I say that with 95% probability, that calculation of that 95% takes into account that these can happen by chance. In any study. So, when I make a statement that these computers made children better at math, I am making the statement with 95% probability made the children better at math. And when I say that, the calculation of what would constitute whether it's 95% or 93% or 97% will take into account the sample size--the number of computers, whether 50 or a million. I understand. I'm just wondering about the fact that there are these other underlying variables. |
| 18:02 | On to some of the findings. One of the fascinating things about the book that I really enjoyed. There's this famous dialog between Fitzgerald and Hemingway: I think Fitzgerald says, The rich are different from us; and Hemingway says, Yes, they have more money. One of the lessons of the book for me is that really isn't the only way to think about it, or a productive way to think about it. There are so many surprising behaviors and unexpected behaviors in these results and other findings that come from treating the poor with a little more care than they act like our models expect them to. I thought we'd start by talking about hunger. Some have argued that hunger traps the poor; if you are really hungry, you can't work very effectively, you can't be productive; if you can't be productive, you can't make much money, so you are going to be hungry. And so the poor are trapped in this cycle. What have we learned about how we react to the opportunity to get a little more food and how that affects their work productivity. I think it's very well established that when the poor get a little bit more money, they spend a bit of it to get more nutrition, but not a lot of it. They spend a bit of it to get more tasty food and a bit on other things. So, they don't act as if their one goal in life is to eat more, get more calories or protein into their bodies, and therefore become more productive. It's very clear that that is not what they are trying to achieve. As a result, if you look at the effect of extra money on extra productivity through this nutrition channel, I think people's view is this is relatively weak. Just by giving people a little more money we won't make them so much more productive that they could then pay that money back. It won't happen. A very optimistic idea would be you give these people more money; they go eat some more chickpeas and get stronger; then they make so much more money they can pay back the money you gave them to buy chickpeas. Mostly, they just don't buy chickpeas. They don't buy high-nutrition foods necessarily--chickpeas or soybeans, cheap high-nutrition foods. Why not? I think a lot of people assume that if you are near starvation, obviously you are going to channel that income into food because that's the biggest problem you have. But as you point out in the book, you tell that great story about the Moroccan person who was pretty hungry but he had a television. I think the poor recognize that even if they behave in the "most rational" way we posit for themselves, eat the chickpeas and all that, they are not going to get that much richer. Basically, this idea that I should postpone all pleasure for a little bit more money in a few weeks is just psychologically unrealistic. You are under a lot of stress, not a very pleasant life; you have this choice. If somebody came and offered you an opportunity to get really rich, said: If you dig this ditch for five days we'll give you a permanent government job with a good salary for the rest of your life--I think most of them will jump to that offer. That's not what is being offered. They are being offered a little bit more: maybe if I eat all the chickpeas I will get 10% more, and then if I keep eating those chickpeas every day I will get 10% more. Is that such a great deal? You are already poor and you have got this little extra money and you could have more fun for a couple of days; maybe that will keep you psychologically more able to deal with all the pressures you are under than having those chickpeas. It's not at all clear to me that if you took their psychology seriously that we would think that it always makes sense to go buy those chickpeas. You are really under pressure, haven't had much fun for a long time, worrying about all these problems all the time--may just want to do something relaxing for a little bit. I don't think that's any different from any of us--just have to make hard choices. You talk about how boring life can be in a small Moroccan village; the person you interviewed said television is better than food. Just a different way to get something out of life. You don't want to die; you are not starving to death. Reminds me of the issue in America where you give a homeless person money on the street and somebody says to you they are just going to waste it on alcohol; my view is if you are homeless on the street, sometimes a beer is a really great thing. Their life is hard; why would you deny them a little bit of pleasure because you think they should have a good meal that day? Just one meal, not life-changing. A little more nutrition or a little more thrill, and you might go for the thrill. Totally rational. Yes. I'm totally with you. I feel like if I were in that position, I don't know why I would be expected to make these kind of morally high-ground choices always. |
| 25:09 | Having said that, one of the interesting issues you discuss across a number of different examples that's so fascinating is opportunities as, say, farmers or parents where there is a technique available that could have what we would say a big bang for the buck. A huge rate of return. It might be using fertilizer for a farmer; for a mother it might be using hydration therapy to keep a child from dying from diarrhea or at least keeping them healthy. And yet often poor people in these countries don't avail themselves of these techniques. Talk about the challenges that the poor face in terms of information and trust, and how different their lives are. I think you kind of anticipated my answer partly. I think we kind of assume that when you tell someone this is what will make your child better--you take oral rehydration therapy and this will obviously make your child better--I think our presumption is that that sounds to them as it sounds to us. One core assumption there is they understand the process by which scientific knowledge is arrived at. We actually don't understand why oral rehydration therapy works much better than antibiotics than diarrhea. Maybe you do, but I don't. I'm not an expert in biology. I don't; that's okay, carry on. I think that's important. But the reason why we believe it is not because we know the science--I don't know the science--but because we know how the institutions which make such recommendations work, how they come up with these recommendations: They do RCTs on these recommendations and they find out this works better than that and then they come up with a recommendation. We know the process of knowledge generation; we have a faith. It's that we believe in a particular process of knowledge generation, we believe that process is more reliable than other processes. Now think of it from the point of view of poor people. I think the fundamental thing that is different is they don't understand the process of knowledge generation, because they never had the education. So they don't understand that there is a Food and Drug Administration (FDA) that does RCTs. So, for them this is just some knowledge that someone is claiming and that sounds counterintuitive. You mean water and sugar is better than a medicine injected into the body? Sounds insane. Medicines are good. Water and sugar is water and sugar. Ultimately oral rehydration is just water and sugar and salt. Why should that work better than the medicine? So, their reaction is: No, we shouldn't give them water because when they have diarrhea these kids are losing water all the time--which is why they die. If we don't give them water, they'll stop having watery bowels. So, they stop giving them water and go give them an antibiotic instead. Their logic--internally it's insane. They are making up a theory of the world from things they understand. In a sense they are less trusting of authority than we are. We can understand the process if a regulator at the FDA decides this or World Health Authority (WHA) recommends that. We kind of know what the WHA is. For the poor, I think they have often been told lies by the state; the government has told them things that haven't happened. So they are more skeptical of institutions generally. So, if you go and tell them things that are counterintuitive, like don't give them the medicine, give them this bottle of water with sugar and salt in it, that's one of the stories that these authorities make up; but God knows why they make those stories up. Or maybe they even think there is some evil plan there; and they should just go with their instinct. Medicine is better than water and sugar. If you state it like that, doesn't it sound plausible to you? Yes; not only that--a lot of times things we rely on and think of as totally scientific turn out to be not as scientific as we had thought. Skepticism is a good idea. The problem is we on the outside often see things we know have been tested, so consistently found to be effective that it's a "no-brainer." And yet children are dying because these techniques are not trusted. Part of that is an education problem; part is a worldview problem; part is as you say, unfamiliarity with the scientific method. And some of it is a distrust not just of the government but of strangers--people who come in from the outside saying, I know what's best for you. It's good to be skeptical of those folks. I agree with you. So, I think it makes perfect sense to react in this way. It's tragic but it's not stupid. It is based on some reasonable thinking. |
| 31:21 | Let's turn to the role of education, which is often thought to be the biggest single barrier to development; and as many have pointed out, there have been some dreadful results and attempts to improve education. This isn't only true in developing countries; it's true in developed countries as well. I always find it ironic that people look at the ineffectiveness of government spending on education via aid and forget how ineffective American spending on education often is in our own country. Let's talk about the big picture first. You contrast what you call the demand wallahs and the supply wallahs, where wallah is a term that means "provider of." Some people suggest we need to build more schools, create more teachers in these poor countries. Others say that unless there's a reason to go to school, a reason to invest, all the schools and teachers in the world won't matter. Talk about what's true and false about those views and what we do know that can make students better educated in poor countries. We know that the demand method--when it's clear that there are benefits from education, people put more effort into it. Also the supply method--when people have schools to go to, they learn more. It's not true that you can just tell people that you should want education and then magically schools will appear. There is a lot of clear evidence that school construction and more generally making schools more available does affect the educational level. Having said that, the reason why that debate is a bit besides the point is that if you look at where the big failures are, they seem to be inflated with our putting effort into sending these children to school and schools are there and still true that people aren't learning. That's the most striking thing. The striking fact is the lack of learning in settings where there doesn't seem to be any obvious lack of demand, or any lack of supply in the sense of there being a school, a teacher, etc. I think what makes it really interesting is that fact. You see a lot of private schools in developing countries. Interesting there are private schools that the poor send their children to, $1 a month or $2 a month kinds of private schools. They are all over the developing world. These parents are very poor and for them, $2 a month is a lot of money, so they are putting effort into this. They are making sure that their kids are getting something that they value. Yet even in those schools, you see slightly better results than the government schools, but still very disappointing results even in those cases. The average kid there isn't studying at great levels, either. The problem seems to be attitude of the entire education system--the teachers, the parents, and even the children toward what the goal of education is. They seem to have the idea that the goal of education is to get through some difficult exam and get some job having got through that. And that's something that only a few people can do it, what we call a winner-take-all education. It's like education is some long large gamble, something everybody should try, but it can't work out for everybody or for most people. Whereas in fact we know that most people get something out of being educated. Even if you can read a little bit you understand what these instructions are from the doctor; you do a little bit better in bringing up your children. The benefits seem to be much more widespread than people assume. The teachers seem to assume that most of these kids are hopeless; there's no point to trying to teach them anything. They just teach to the top of the class. And the parents don't complain, because they also think that the goal of the whole system is to train somebody, to find out whether your child is one of those few lucky geniuses who is going to go on to get a good education and a fine job; and if he isn't, what's the point of educating him? Everybody is much more pessimistic about the education outcomes of the median person than they should be. So, based on that, the curricula are way too hard, the teachers don't pay any attention to any of the children falling behind in class, the parents don't complain when the children fall behind in class--either they assume that there is some rough justice there, either your kid is really smart and the education is worthwhile, otherwise there is no point. So, everybody kind of colludes with that, and the kids very quickly lose hope. They kind of figure out that they are not one of these anointed people, and they start giving up. You see these children sitting through class after class where they understand nothing. They are in fourth grade, they can't really read. They are teaching history, they understand nothing of what's going on, but they sit calmly through school and start kind of dropping out. They vote with their feet. When they are in fourth grade they are too young; maybe keep coming. But by the time they reach sixth or seventh grade, they know the school thing is not working out for them, so they just drop out. You see this pattern over and over again, of unreasonable expectations that then are effectively used to clean out most of the people in the education. |
| 38:27 | You chronicle as many others have a huge problem with absenteeism by teachers, which is independent of this attitude problem you've mentioned. I'd like you to talk about the magnitude of the absenteeism problem, what might be done about that, and then what might be done about this attitude issue that the whole thing is a lottery for the elite. I think the magnitude is shocking sometimes. Roughly in many countries the actual teaching time that you get from teachers is about half of what they should be delivering, because a quarter of the time they don't show up and a quarter of the time when they show up they don't teach. If you take those two together, it's about maybe 50% of their regulation teaching time, they don't actually teach. I think that's a huge problem. A problem in some ways we know. What can be done about it? Some of my colleagues from the Poverty Action Lab, including Esther Duflo, my co-author on this book, did this experiment where they induce, actually had people teachers who would show up 60% of the time. This was in the old days before the information technology actually hit the camera. Still in the days when you had film cameras, cheap film cameras given to every teacher; and the teacher was told: at the beginning of the day, have a child take a picture of you with at least 8 other children--minimum class size--and at the end of the day do that again. And the camera had a date and time. The arrangement with the teachers was that they would be paid a fixed amount plus a pro-rata amount based on the number of days they attended, and this date and time stamps were just to give proof that they had attended. So, this was proof; based on having the proof, they would get paid more for every day they attended. You look at what that did--it halved absence rate and the children learned much more. Channel there of how to improve education. It is something straightforward to do if there is political will to do it. I don't think this is a difficult problem, as such. These days with fingerprint, you could automatize the system with very little cost and make it much more effective than this. This was a system that was designed for the world ten years ago; now you could do it much more effectively using information technology. So, that's the easy part. Then there is a hard part, which is how do you get teachers to have different attitudes. That is not easy but not actually as hard as it seems. One thing that is kind of heartening is that when the same teachers, the ones who teach in government schools and ignore most of the children and act in this elitist way--when you train them to teach in a summer school for the lagging children, the lagging children actually learn a hell of a lot from them. The teachers are very effective in teaching the lagging children. It's striking that the same person who actually ignores the lagging children when he teaches in regular school seems to be very effective in teaching those children when in summer school that's his job. So, it's a matter of the education system sending the signal that this is what we want. It's not a huge battle to be fought. Some teachers have obviously checked out; but they are willing to try to teach. It's just difficult for them to try to teach these lagging children in the regular school system because the whole school system expects them not to do that. Interesting challenge. It's a challenge in America, too, where there is I think a surprising lack of oversight in the public school system. |
| 44:11 | One of the things it reminds me of, that I think was so interesting in the book, is how many poor people in poor countries aspire toward government jobs, and often as teachers. This is a little disheartening, although when you think about the opportunity for absenteeism and drinking tea, which is one of the things you mention these teachers often do--even when they are in school they often are hanging out drinking tea--you can understand why a government job is so attractive. And often the private sector in some of these countries is not very well organized. For a variety of reasons, governance being one of them, it's hard for the private sector to generate employment through legitimate channels. Talk a little bit about what hope we have in that area. I think that in general the problems of the private sector in developing countries is interesting. You see huge numbers of private sector stores--you see how many small stores there are on the street, how much private enterprise there is in these countries. Also true that if you look at the productivity of the private sector, it seems very low. I think that sort of poses an interesting challenge: Why are all these people in the private sector if the private sector is not very productive? Turns out, not surprisingly, the reason is the private sector is not productive. Because there are no good jobs around, people substitute by setting up their own little firms and trying to eke out a living by selling what they can sell or doing some service that somebody may want to buy. So you end up with a huge number of inefficient firms. What all these countries need to work is to have much bigger, more productive and less numerous firms. That's the big challenge. We've been told this is going to happen from microcredit; but I think microcredit is the wrong instrument to generate large firms. Second, loans have the structure where you are required to pay back the loan in the weeks after you got the loan. You pay back a chunk of money every week, so you are really stuck. You have to get your loan to generate an income immediately. That doesn't make for being really an entrepreneur in the adventurous sense of the word. You just find the easiest, simplest, most reliable thing you can do which you can start tomorrow and generate some money immediately and you do that. Very unlikely that you are going to become big and productive. |
| 47:43 | Suppose you are listening to this podcast; you care about the poor; you'd like to do something. What can someone do as an individual to help people around the world? One answer would be to lobby the government to give more foreign aid. That's not generally what people think of first; and it tends not to be effective when it's just spent. What can an individual do to help? I think there's a lot they can do. I think there are many, many good organizations out there doing good work. We list some on our website. But I think the more general lesson we would like people to embrace is a). not to lose hope--there is lots of good stuff happening. It's not that every effort fails; lots of good solutions, some of them even in government, contribute to welfare. But I think the important thing is to have a critical judgment, to ask the question not glibly and not believe in heartwarming thoughts despite all evidence. I think people need to be more active consumers, need to think why this would work, what's the evidence, what's the story they are telling; why do I believe that story; is there any evidence? |
READER COMMENTS
Julien Couvreur
Jul 11 2011 at 11:29am
I wasn’t quite convinced by the question of methodology. Let’s say that 40% of schools have some favorable condition (say better teachers), and the other 60% doesn’t, but that is treated as an unobserved variable. Let’s say that the software makes not differences. Isn’t the unknown distribution of the unobserved variable likely to skew the result after random draw of 50 schools?
What happens when more variables are introduced (does that help or hinder the experiment)?
If you don’t know those variables and distributions, how do you decide an appropriate sample size?
Mads Lindstrøm
Jul 11 2011 at 1:33pm
I am not a statistician nor particular knowledgeable about statistics. That said, randomized controlled trials is, to the best of my knowledge, what doctors most commonly use when testing medicine. Therefore, it seems safe to assume, that the randomized controlled trials method has received a lot of scrutiny. I am not saying that people should not question the technique, but we can reasonably assume that most basic questions have been answered.
Also see http://en.wikipedia.org/wiki/Randomized_controlled_trial#Statistical_error .
Personally, with my limited knowledge, I trust randomized control trials more than when people control for X, Y and Z. The problem is, why X, Y and Z? Why not control for variable A and B? Maybe X is really an effect of “the computers in the classroom”, rather than something we need to control for (only applicable when we don’t have data, before putting the “computers in the classroom”).
Rucksack Revolution
Jul 11 2011 at 4:48pm
I am not a statistician either, but maybe I can better explain what Abhijit Banerjee was saying about the methodology of the RCTs.
Let’s say that teacher quality is one variable that varies between schools. This variable will thus affect the dispersion of results within each sample. In other words, if all schools within the sample were basically the same, the dispersion of results would be fairly narrow around the mean. If they are not all the same and there are significant individual differences, the dispersion will be greater in both the computer and non-computer group.
The researchers will then use statistical analysis that incorporates the mean of each group, the dispersion around the mean (standard deviation) of each group, and the sample size of each group to see how likely it is that this result is due to chance alone. Here, “chance” includes differences among individual schools in things such as teacher quality. So the fact that there might be various unintended differences between groups A and B will be taken into account in the analysis by the measure of variance known as the standard deviation. If the variances of group A and B are different (which I think is what Russ is suggesting), then I believe that can be incorporated into the statistical analysis. Maybe someone can elaborate…
xian
Jul 11 2011 at 10:21pm
statistics/methodology:
it’s a t-test analysis. if any property/measurement of the samples’ fail the t-test, then they can not b said to represent a random population (for the given size of the sample).
Floccina
Jul 12 2011 at 9:13am
I love what he said about education. It seems to me that school does not teach enough practical valuable information and skills for the students lives. School to me looks more like a long test. It seems that testing function of schooling has squeezed out education. For example why not teach farming cultures better farming techniques?
Arnim Sauerbier
Jul 13 2011 at 4:35am
Please give me a link to an explanation of Russ’s objection to the claims of the statistical approach. I have a basic college statistics background. Thank you.
xian
Jul 13 2011 at 11:33am
as far as i can tell, russ’s objection was pretty handwavy…kinda skeptical for it’s own sake, but think there was honest intention to at least to raise the issue so as to remind people that u should always b mindful of statistical methods.
this is probably bc the interview wasnt supposed to b about methodological minutiae.
Russ Roberts
Jul 13 2011 at 1:33pm
I raised the statistical issue as an aside. I was surprised by the answer–I had assumed he would say something else–so I didn’t do a very good job pursuing it and then dropped it because there was so much else to talk about.
What I was trying to say is that when you take 100 villages and randomly assign 50 to a treatment of some kind (a computer tutorial for example) and use the other 50 as a control group, I though you would want to see whether the group of 50 in the control group were similar to the 50 in the experimental group. That is, you’d want to be assured that the random sample was representative and that any random variation in the results was due to randomness in how the students performed on the test.
Think about a true laboratory experiment to see if the chemical cyclamate causes cancer in rats. So you have a group of rats whoget dosed with cyclamate and a group that doesn’t. The bigger the sample, the more confident you are that the differences in cancer rates between the two groups are due to the cyclamate as opposed to random variation. But when you do a test like that, you want rats that are similar in genetic makeup. You don’t want to take 50 of your rats from a group that grew up in the woods and compare them to 50 rats that grew up next to a nuclear powerplant. You’d be worried that some of the differences in the cancer rates would be due to other differences between the two populations other than the treatment.
So what I was surprised about was that they don’t seem to do this in the development literature. If the 50 villages who get the treatment have a much higher IQ or better educated parents or better nutrition, then how would you know that the computer tutorial was effective rather than these other factors? Yes, as the sample size grows (to a million, say) these other factors are very very likely to be the same across both groups. But in a smaller sample, don’t you have to worry about this? Or don’t you have to take this into account when you calculate your confidence interval around the size of the effect of the treatment?
I may be totally wrong here. Would love to know. I’ll try to get some better educated econometricians than myself to weigh in.
xian
Jul 13 2011 at 2:08pm
from the paper “The miracle of microfinance? Evidence from a randomized evaluation” experimental design section:
“In each area, a baseline survey was conducted in 2005. Households were selected for the baseline survey conditional on having a woman between the ages of 18-55 in the household.
Information was collected on household composition, education, employment, asset ownership, decision-making, expenditure, borrowing, saving, and any businesses currently operated by the household or stopped within the last year. A total of 2,800 households were surveyed in the baseline.(4)
After the baseline survey, but prior to randomization, sixteen areas were dropped from the
study. These areas were dropped because they were found to contain large numbers of migrantworker households….”
it goes on…i think this means they’re trying to control for stuff like that. and then there’s caveat about census taking (a big one it seems)
“(4) Unfortunately, the baseline sample survey was not a random survey of the entire area. In the absence of a census, the Örst step to draw the sample was to perform a census of the area. The survey company did not survey a comprehensive sample, but a sample of the houses located fairly close to the area center. This was rectiÖed
before the endline survey, by conducting a census in early 2007.”
which they say they address, so im guessing they mean it.
Krishnan
Jul 13 2011 at 2:46pm
I understood the statistical issue this way – What he said was that if 50 schools were indeed picked AT RANDOM from a list of 100, then statistically each of the 50 schools can be expected (at some level) to be “similar” and that you can calculate what the difference would be between these two groups – then do an experiment (i.e. give one group say a tutorial program) and calculate the difference again – and statistics allows one to calculate if this difference BECAUSE of the program is statistically larger than without the program
What he did not address was – whether the 100 schools were a true representative “sample” of the entire “population” of schools – and if the sample is truly representative of the population, then yes, the analysis can help
A key in such studies is to make sure that the sample is indeed representative of the population – For example if they took 100 schools from a big city and 100 schools from say some rural area and if the effect of the introduction of the program were statistically significant, it would add support to the hypothesis that that program was beneficial
A BIG problem with almost any study using humans is the difficulty of selecting a “representative” sample from the “population” at large – and more importantly have some data indicating as to why the sample is indeed representative of the population at large
Krishnan
Jul 13 2011 at 2:55pm
On the issue of parents in many countries having difficulty understanding that water with salt and sugar may be better than drugs for diarrhea… Banerjee’s conclusion had to do with how such people do not understand modern science/conclusions/studies … they cannot understand why drugs would not be better
We have a similar problem with many parents in this country (and other developed nations) – people popping antibiotics for the common cold for example even as we have known for years that it does not do any good – and the abuse of palliatives with children when they have colds/coughs – It seems that no amount of information/analysis is sufficient to convince many parents that palliatives are not cures and can cause more harm than good … It was only very recently that the FDA issued warnings about the use of over the counter drugs for many common ailments for children
The idea that for many things that afflict us, often the simplest solution may be best and does not need complex chemicals is lost on many – inspite of living in highly developed countries.
There is something about a “drug” or “medicine” that makes many think they are indeed “magic” bullets for anything that ails
bn
Jul 13 2011 at 3:33pm
I think Russ’s statistical objection is related to the fact that the more facets a die has, the more tosses one needs until it starts to approach the expected outcome. A classroom full of kids is an extremely multifaceted thing. 50 tosses is barely enough to approach the expected outcome of a coin flip. No?
xian
Jul 13 2011 at 3:33pm
i really think it’s the t-test or some variation/application of it. that’s the tool(s) to use here…it’s been done before
http://www.socialresearchmethods.net/kb/stat_t.php
the only question is if was applied correctly, which would take a bit more spadework.
AHBritton
Jul 14 2011 at 12:49am
Russ Roberts & Julien Couvreur,
I think you might be confused about the issue at hand. You seem to be asking “what if there is some systematic misrepresentation in either the control group or test group?” The key here is not really a test of how the statistics work, but seem to be a skepticism of the sampling. If a reasonably representative and random lot can be picked than everything Banjerjee said would logically refute your skepticism.
Using the coin flipping example. Let’s say you decided flip 100 random coins 100 times, but before flipping you decided to designate the first 50 as the “control” and the second 50 as the “test group.”
Now after flipping you discover that the “test group” was entirely heads. The odds of this are so unlikely that it would almost certainly represent a deliberate tampering with the experiment while in progress. Why? Because EVEN IF the initial coins were rigged so that 50 of them would always land heads, if they were randomly chosen they statistically would be highly unlikely to all land on one side of the sample. It is true that the sampling would be distorted, but it would be distorted evenly. It would require BOTH the rigged coins AND a rigged selection process to truly create a lopsided set.
Similarly, EVEN IF the sample set was say overly represented by city dwellers, this would only in theory effect the scope of the conclusion, not the conclusion itself. This can be easily fixed by taking pains to truly choose a representative sample population from which to draw.
Russ, as you said, you would be convinced by a sample size of a million, but why? The only thing that has changed is the probability of accidental distribution errors, but as Banjerjee explained, this is addressed by the fact that they present the probability up front based on the sample size and other statistical factors. That is basically like saying 95% probability is not good enough, I need a 99.999999999% probability before I’ll draw any conclusions.
If this was the case, almost all fields of inquiry would likely grind to a halt. Most people would be satisfied with 95% probability, at least for drawing a tentative conclusion.
Jonathan
Jul 14 2011 at 8:33am
Russ,
Yes, researchers usually check whether treatment assignment was orthogonal to baseline characteristics. No, they don’t need to add control variables to address this issue; the expectation of the difference between two randomized groups is zero and the variance of the difference is proportionate to the variance of the individuals, so as the chance of a “unbalanced” random draw increases, so does the standard error of the estimated treatment effect.
You add control variables to increase power:
“In general, controlling for variables that have a large effect on the outcome can help reduce standard errors of the estimates and thus the sample size needed. This is a reason why baseline surveys can greatly reduce sample size requirement when the outcome variables are persistent. For example, controlling for baseline test scores in evaluations of education interventions greatly improves the precision of estimates, which reduces the cost of these evaluations when a baseline test can be conducted. Note, however, that controlling for variables that explain little or none of the variation in the outcome will increase standard errors by reducing degrees of freedom. Choosing which variables to control for is therefore a difficult exercise. Note that this choice must in principle be specified in advance to avoid the risk of specification searching.” Duflo, Glennerster & Kremer – Using Randomization in Development Economics Research – A Toolkit (CEPR 2007)
Martin
Jul 14 2011 at 9:05am
Hi,
I read an amusing book on randomness
“The Drunkards Walk”
I speaks about this problem and is a fun read as well.
Speaks about the law of large numbers and this problem.
Russ Roberts
Jul 15 2011 at 2:36am
Ed Leamer directs me to his 2010 paper from the Journal of Economic Perspectives, Tantalus on the Road to Asymptopia, a reaction to a paper by Angrist and Pischke (which in turn is something of a reaction to Leamer’s 1983 “Let’s Take the Con Out of Econometrics.”
The relevant passage is from pp. 33-34 in the JEP pagination under the heading “Randomization is Not Enough”:
Leamer goes on to discuss other issues related to Randomized Control Trials raised in work by Heckman and Deaton. We will put up links to these papers. Banerjee and Duflo respond to some criticisms and defend their approachin this paper–we’ll add this paper as well to the formal links section.
xian
Jul 15 2011 at 9:31am
nice….
nearly all of the comments have been on this topic.
kinda makes me think this is an interview topic in itself…perhaps a couple of episodes so everyone has a chance to address the concerns of the other.
keatssycamore
Jul 15 2011 at 1:35pm
Mr. Roberts,
Your above link to Leamer’s paper does not work (the Banjaree link is fine), maybe Lauren can fix it?
Russ Roberts
Jul 15 2011 at 2:08pm
Here is the correct link to the Leamer paper referenced in my comment.
Demaratus
Jul 15 2011 at 3:45pm
Is Banarjee making any assumptions of a Gaussian normal distribution about the sample schools in his example? Or does his logic hold regardless of the distribution?
Frank Howland
Jul 16 2011 at 9:55am
That 2:08 PM July 15 link to the Leamer essay didn’t work. Try this:
http://www.anderson.ucla.edu/faculty/edward.leamer/selected_research/ucla_anderson_faculty_edward_leamer_selected_research.html
neil pelkey
Jul 22 2011 at 3:00pm
Dear Demartus,
Type II error matters a lot in policy choice. Analyzing small samples with the incorrect distributional assumptions will often give incorrect answers in w.r.t. policy. False negatives can abound leading to the inability to differentiate high performing and low performing choices–which is the poit of RCT after all.
See
http://faculty.juniata.edu/pelkey/negbinanova2macro.xlsm
for an excel sheet where you can play with the impact of using the wrong distributional assumption and false negatives for a smaple size of 30.
you will need:
http://www.poptools.org
It does not really matter which distributional assumption you pick. If it is wrong even so-called huber white will save you.
Prof. Banerjee is correct if the sample sizes are quite large (>100 per treatment category) but quite large enough may be well beyond all but the most highly funded researchers. Sample sizes of 20 – 30 are not quite “large” enough.
Randomization is a good thing, but it is no more a panacea for policy choice than microfinance is a panacea for poverty.
For a rigourous treatment see:
Inmaculada B. Aban, Gary R. Cutter, Nsoki
Mavinga, Inferences and power analysis concerning two negative binomial distributions with an application to MRI lesion counts data, Computational Statistics & Data Analysis, Volume 53, Issue 3, Computational Statistics within Clinical Research, 15 January 2009, Pages 820-833, ISSN 0167-9473, DOI: 10.1016/j.csda.2008.07.034.
(http://www.sciencedirect.com/science/article/pii/S016794730800371X)
Comments are closed.