In cases in which a specific antecedent (e.g., an instruction) are presented, trial-by-trial interobserver agreement (IOA) may be useful. Trial-by-trial IOA measures the occurrence or non-occurrence of a specific response following the presentation of a discriminative stimulus.
For example, observers might record a “+” every time a child stands up after being asked by an instructor and a “-“ when the child does not stand up after being asked by an instructor.
Trial-by-trial IOA percentage is calculated using the smaller number as the numerator and the larger number as the denominator, as with total count and mean count-per-interval IOA (see previous post).