Spencer SCD @ Vandy Part 2: Functional Relation

I have misunderstood the “functional relation” and potential implications for meta-analysis

Prior to my time at the conference, I had understood the functional relation as being case-specific, related to the basic effect demonstrated by a case or phase pair. Before, I thought you had a functional relation for an individual whenever you had an observable basic effect that was “big enough” to be considered meaningful. Instead, it quickly became clear to me during the conference that the functional relation is related to both the “big enough” basic effect for each case or phase pair and the demonstration of the three replications in the study or design. Wendy also suggested that the functional relation might not just be a function of, say, an individual treatment/reversal design. If the study has three or six treatment/reversal designs, then the functional relation might be considered to be demonstrated not just by the individual designs but also the replications across the designs in the study.

This is a little embarrassing because Wendy Machalicek gently pointed out in a lunch conversation that the present broad understanding of “functional relation” is discussed in the pilot SCD standards which contained visual analysis. I’ll admit to generally being a skim-reader in that I tend to skim text until I hit the part that’s relevant to the specific question I’m interested in. The last time I read the pilot standards beginning to end in detail was a several years ago, and I was worried about things other than what exactly was meant by “functional relation” at the time. I have looked at parts of them a bunch of times in the past few years in my work with the WWC, but never to clarify my understanding of the functional relation because I thought I already understood it.

I think this understanding of a functional relationship has some implications for meta-analysis, or at least I have some questions. Does a meta-analysis itself potentially demonstrate a functional relation? It’s a collection of designs related by populations, outcomes, and/or interventions. This reminds me of a conversation that James P. and I once had where he suggested that perhaps the WWC should consider meta-analyses of SCDs a “study” for the purposes of the review. I thought it was kind of odd at the time, but maybe not so much now.

Moreover, can you in some sense rescue designs which are otherwise not useful for causal inference (say, an AB design, or a multiple baseline with only two participants) using meta-analysis? Something like an AB design is still a causal design, it is simply hopelessly subject to unobserved confounds by itself. If you’ve got enough of them, and you perform meta-analysis, do you have a reasonable estimate of the causal effect? Thanks to Joe Lambert for making me think about this. As an aside, the vagueness over what exactly constitutes a functional relation worries me a little bit. I’m not sure if I’m worried for good reason, because I can’t really describe anything other than formless concern. Maybe just because wherever things are vague in research design or analysis, I start to worry about unforeseen consequences.