I still don’t know what stability is or how important it is
Jennifer Ledford opened the conference with some important details about the structure and origins of the conference. We were arranged across the rotunda in small tables that seated 8-10, and Jen asked us to talk to our groups about what we thought the big questions or problems were that remained in SCDs. John Ferron and I were sitting at the same table, in no small part because we mostly only knew each other and no one else, and we both immediately tried to ask the same question. What exactly does it mean for a baseline to be stable in SCDs? When SCD authors talk about what drove the choice to intervene at a particular time (mostly they don’t say) it’s common for them to say “we started the intervention when baseline data were stable” or something to that effect. Almost never do they describe what features of the data drove that decision.
Texts like Jen’s have sections that talk a little bit about the decision to intervene and possible decision rules, but I’m skeptical that the real decision rules are captured there. Not the least of which because I don’t think visual analysis is any one thing but represents a set of related practices that mostly (but not completely) agree. More than that, when I was discussing my problem with Tim Slocum and Wendy I articulated something that I hadn’t been able to articulate before. I suspect that many visual analysts make this choice as a function of practice. They’ve built a decision-making engine in their head that makes choices that are probably reasonably reliable within researchers and across researchers from similar traditions. But they aren’t really making a decision based on a formal decision rule. I think Tim called this the difference between “contingency-based” and “rules-based” behaviors. When you’re asked to describe why you did a contingent behavior as if it were a rule-based behavior, you’re liable to make something up that may or may not relate to the real decision. Related to what I’ve already talked about above, I think that stability can be useful, or at any rate not harmful, when researchers are engaged in inductive research practices. I’m already skeptical that we should be estimating effect sizes from an inductively oriented study, and so the consequences for statistical models applied to response-guided data are maybe irrelevant if you’re estimating effect sizes for the “correct” studies. When you want to estimate an effect size or perform some kinds of hypothesis tests, response-guided data can maybe get you in trouble. Without a really good understanding of what constitutes stability, it’s hard to know what those consequences are, other than that the vague threat looms.
Therapeutic trends in baseline: an issue?
One thing that seems common in response-guided designs is avoiding a therapeutic baseline phase (this is frequently some element of stability). I have thought about this a lot in the context of effect size estimation and my work on the potential impacts from response-guided designs in SCDs.
However, there are several reasons for an interest in therapeutic trends in baseline that aren’t necessarily related to analysis but have more to do with the choice to intervene at all. I have heard applied researchers say that they don’t want to intervene when kids don’t need it. If the baseline is therapeutic enough the researcher might not just wait until the baseline shows evidence of a pattern of problematic behaviors, they might decline to intervene at all.
There are sometimes ethical reasons for this. The intervention might be aversive or even harmful for a student who doesn’t need it. It also might be costly in time and effort for the interventionist, the student, and maybe the family if the student doesn’t need it. However, one thing I haven’t seen discussed as much is that the baseline is an artifact of research! The kids in these studies are not picked randomly, they are selected because someone in their lives (probably parents and/or teachers) think that something is bad enough that help is needed. Absent the need to establish a baseline level of functioning to demonstrate a functional relation, an interventionist (or a teacher or parent who is using an intervention supported by SCD research) is not going to wait to intervene!
This means that some proportion of children who would receive the intervention in real life are missing from the SCD literature. If you’re an interventionist, I think you should be asking yourself “What happens to students who don’t need this intervention but receive it anyway?” If the outcomes would be negative, there are contexts where this won’t matter. Most thoughtful applied interventionists will probably cease an intervention that appears ineffective or even harmful relatively quickly. But maybe not all interventionists, especially in contexts where “it gets worse before it gets better” as I heard mentioned a couple of times at the conference. “Naïve” interventionists (teachers, parents etc.) in particular may persist in the face of an iatrogenic response. This seems like a problem! Understanding the responding of individuals who don’t need an intervention but might receive it in practice seems like something that we want in the intervention literature. Although group design work doesn’t necessarily reflect individual negative responses very well, it does contain any aggregated iatrogenic effects in the effect size.