This is a follow-up to a previous post on unmeasured interaction variables in social science. Last time I asked: If there might always be some unmeasured interaction variable that could change the causal direction of a finding, on what basis can we ask that reasonable people believe the finding applies to a specific case? The answer was highly pragmatic. Knowledge is contingent. On average, betting on solid findings will beat hunting for exceptions.
But what else should we ask if someone proposes an unmeasured interaction variable as a reason not to expect a social science finding to apply to a particular case?
It seems to me we can ask questions of the proposed interaction variable, of the person proposing it, and of the finding.
Of the finding we might ask — beyond the specifics of the given study, which I’m assuming is rigorous — how well understood is the underlying phenomenon or area of inquiry, in general? Subjects that are generally well understood — where the majority of variation is reasonably well explained both empirically and theoretically — are worse candidates for hidden interactions, in general, than less well explained ones.
Of the speaker, we can ask the typical Tetlock-ian questions. What is their belief accuracy? Failing that, do they generally seem to display good habits of mind, including frequent deference to empirical findings? In other words, they have more authority to propose a hidden interaction effect if we know they are usually right about things or, at least, if they usually do bet on solid research findings.
Then, finally, we can ask questions of the proposed interaction variable itself. First and foremost, we might ask: has this variable been shown to matter in related contexts? We know sleep affects cognitive functioning and performance, for instance. So sleep is a more plausible hidden interaction variable in the context of cognitive functioning than, say, the color of your t-shirt. Relatedly, we can ask, as that example suggests, how plausible the proposed causal mechanism is. Usually, though perhaps not always, these will go hand in hand. Proposed causal mechanisms for a hidden interaction will be more plausible in cases where they’ve been shown to matter in a related context.
beyond the times 