During the last century, the social sciences have started to focus much more on internal validity, e.g., identifying a causal effect as well as possible. This went to the detriment of other factors like external validity and to a focus on relatively simple causal relationship, including only a small set of variables, because only they can be causally identified well.
I guess that this strong focus has selected and formed researchers in a way that makes them less likely to acknowledge the trade-offs you describe, compared to previous generations of scientists. For the progress of science, a focus on internal validity may not be a problem, one could argue whether it is currently too strongly pronounced or not. But when such thinking is transferred to politics, the result is a narrow focus on maximizing one particular variable, at the cost of everything else, which is rarely welfare optimal.
There may something to that. Since running RCTs forces you to specify and now pre-register specific outcomes of interest, folks may now a bit more blind to the broader implications and trade-offs.
Because this is an ongoing concern of ours, I just want to say that this post could have benefited from AI editing. All of your arguments make sense in context, but the overall flow was hard to follow, and indeed a post with such a clear question in the title failed to clearly answer it in the text.
Two reasons for that: one, you smuggle a lot in with the term judgment, which is an important one that I think often bears the brunt of our anxieties over knowledge without being, as the models often put it, capable of bearing the load. It's worth saying what you mean about that concept/activity/capacity.
Secondly, you seem to have selected the title last and not let it structure the post! You could have clearly said: evidence-based policymaking too often assumes RCTs are the only form of evidence that matters, too often encourage expert theater over democratic deliberation, too often smuggle value judgments in with the empirics, too often resolve the narrow causal inference but don't address the broad field of potential remedies. You could also make a whole meal out of the fact that policy goals are often multiple and even mildly contradictory!
In my writing on this I've tried to emphasize the asymmetric demand for gold standard evidence, and you make that point too but not as clearly as I'd like!
GPT might have suggested this if you'd asked, but I wonder if the criticism you've received recently has caused you to rely on it less. Stick to your principles, please!
Thanks, some of this is a fair critique! Though to be completely transparent, I have had the title first, but it was in some tension with my secondary goal of proving a quasi book review to Macedo and Lee which I read a couple of weeks ago but couldn't stop thinking about.
I think the problem is more acute in public health/medicine than in the social sciences. Most people who run RCTs in economics for example will quite openly admit that the results might not generalize (but they chase publications...), but in medicine, their version of statistics training involves showing people this pyramid: https://guides.lib.uchicago.edu/c.php?g=297120&p=1983547
I worry you might be at risk of running together two types of judgement. First, normative/value judgement - things like how to trade-off benefits and burdens between compatriots and foreigners, or children's schooling vs older people's health. Second, empirical judgement - the art of pragmatically synthesising different sources of evidence.
The former is absolutely the domain of politicians, though I think they are far less comfortable in it than they might have been 40 years ago, in a more ideologically structured political era.
The latter is more appropriate for technocrats - and who I mean by that varies, sometimes academics, sometimes civil servants. And it's hard, and I think they vary in their aptitude for it. But I'd trust them far more than politicians who generally have much less experience and ability when it comes to critically evaluating evidence.
I really like this essay, but it almost feels like the second to last paragraph about openness to evidence misses the mark. Isn't the whole point of this article building up to the idea that setting up the breakers of our convictions to loosely defined "evidence" has led to issues among the elite?
For example, it seems far more reasonable to me that someone should be able to show you a logical model or plausible argument to change your mind. Or for a new policy to be implemented and for us to review the cumulative evidence that starts to appear slowly exactly as you said earlier in the article
Thanks, this is a fair push, and I think we mostly agree. But, just to clarify, "what would change my mind" was never code for "show me an RCT."
From my perspective the line between models and evidence is blurry anyway: a formal model is a kind of thought experiment, and empirical evidence is itself a model of the world. It can be both, and good reasoning uses whichever the question allows. The honest move here is just being willing to name what would move you, which is exactly what the experts refused to do in the pandemic example.
I think I'm more getting at the fact that it doesn't sit right for me that the standard to be moved from a position that someone arrives at via intuition/abduction is to observe sufficient evidence.
My experience is that if I intuitively believe something to be true, and then some guy tells me there is some study that says otherwise, it is usually better to trust my intuition. (Not always but mostly)
Immigration is one such thing. How can immigration always be beneficial? That makes no sense. It's like saying we could take in all the criminals from all over the world and still benefit.
Or the transgender stuff. There are so many who regretted their transition that I have seen or read, that the claim it is only 0.2% makes no sense.
During the last century, the social sciences have started to focus much more on internal validity, e.g., identifying a causal effect as well as possible. This went to the detriment of other factors like external validity and to a focus on relatively simple causal relationship, including only a small set of variables, because only they can be causally identified well.
I guess that this strong focus has selected and formed researchers in a way that makes them less likely to acknowledge the trade-offs you describe, compared to previous generations of scientists. For the progress of science, a focus on internal validity may not be a problem, one could argue whether it is currently too strongly pronounced or not. But when such thinking is transferred to politics, the result is a narrow focus on maximizing one particular variable, at the cost of everything else, which is rarely welfare optimal.
There may something to that. Since running RCTs forces you to specify and now pre-register specific outcomes of interest, folks may now a bit more blind to the broader implications and trade-offs.
I just want to say I thought this article very good, very interesting and that it flowed quite well.
Because this is an ongoing concern of ours, I just want to say that this post could have benefited from AI editing. All of your arguments make sense in context, but the overall flow was hard to follow, and indeed a post with such a clear question in the title failed to clearly answer it in the text.
Two reasons for that: one, you smuggle a lot in with the term judgment, which is an important one that I think often bears the brunt of our anxieties over knowledge without being, as the models often put it, capable of bearing the load. It's worth saying what you mean about that concept/activity/capacity.
Secondly, you seem to have selected the title last and not let it structure the post! You could have clearly said: evidence-based policymaking too often assumes RCTs are the only form of evidence that matters, too often encourage expert theater over democratic deliberation, too often smuggle value judgments in with the empirics, too often resolve the narrow causal inference but don't address the broad field of potential remedies. You could also make a whole meal out of the fact that policy goals are often multiple and even mildly contradictory!
In my writing on this I've tried to emphasize the asymmetric demand for gold standard evidence, and you make that point too but not as clearly as I'd like!
GPT might have suggested this if you'd asked, but I wonder if the criticism you've received recently has caused you to rely on it less. Stick to your principles, please!
Thanks, some of this is a fair critique! Though to be completely transparent, I have had the title first, but it was in some tension with my secondary goal of proving a quasi book review to Macedo and Lee which I read a couple of weeks ago but couldn't stop thinking about.
You did great
Always more fun to structure things around your reading list!
Here is a good post by Scott Alexander on the topic: https://www.astralcodexten.com/p/the-phrase-no-evidence-is-a-red-flag
I think the problem is more acute in public health/medicine than in the social sciences. Most people who run RCTs in economics for example will quite openly admit that the results might not generalize (but they chase publications...), but in medicine, their version of statistics training involves showing people this pyramid: https://guides.lib.uchicago.edu/c.php?g=297120&p=1983547
Yep, and given that public health and medicine are more consequential for our lives, this is a big issue.
P.S. I knew there was some other SA essay I forgot to reference LOL
I worry you might be at risk of running together two types of judgement. First, normative/value judgement - things like how to trade-off benefits and burdens between compatriots and foreigners, or children's schooling vs older people's health. Second, empirical judgement - the art of pragmatically synthesising different sources of evidence.
The former is absolutely the domain of politicians, though I think they are far less comfortable in it than they might have been 40 years ago, in a more ideologically structured political era.
The latter is more appropriate for technocrats - and who I mean by that varies, sometimes academics, sometimes civil servants. And it's hard, and I think they vary in their aptitude for it. But I'd trust them far more than politicians who generally have much less experience and ability when it comes to critically evaluating evidence.
I really like this essay, but it almost feels like the second to last paragraph about openness to evidence misses the mark. Isn't the whole point of this article building up to the idea that setting up the breakers of our convictions to loosely defined "evidence" has led to issues among the elite?
For example, it seems far more reasonable to me that someone should be able to show you a logical model or plausible argument to change your mind. Or for a new policy to be implemented and for us to review the cumulative evidence that starts to appear slowly exactly as you said earlier in the article
Thanks, this is a fair push, and I think we mostly agree. But, just to clarify, "what would change my mind" was never code for "show me an RCT."
From my perspective the line between models and evidence is blurry anyway: a formal model is a kind of thought experiment, and empirical evidence is itself a model of the world. It can be both, and good reasoning uses whichever the question allows. The honest move here is just being willing to name what would move you, which is exactly what the experts refused to do in the pandemic example.
I think I'm more getting at the fact that it doesn't sit right for me that the standard to be moved from a position that someone arrives at via intuition/abduction is to observe sufficient evidence.
My experience is that if I intuitively believe something to be true, and then some guy tells me there is some study that says otherwise, it is usually better to trust my intuition. (Not always but mostly)
Immigration is one such thing. How can immigration always be beneficial? That makes no sense. It's like saying we could take in all the criminals from all over the world and still benefit.
Or the transgender stuff. There are so many who regretted their transition that I have seen or read, that the claim it is only 0.2% makes no sense.