Does Task Complexity Moderate the Benefits of Liveness? A Controlled Experiment

Patrick Rein¹ , Stefan Ramson² , Tom Beckmann³ , and Robert Hirschfeld⁴

The Art, Science, and Engineering of Programming, 2025, Vol. 9, Issue 1, Article 1

Submission date: 2024-05-31
Publication date: 2024-10-15
DOI: https://doi.org/10.22152/programming-journal.org/2025/9/1
Full text: PDF

Abstract

Live programming features can be found in a range of programming environments, from individual prototypes to widely used environments. While liveness is generally considered a useful property, there is little empirical evidence on when and how liveness can be beneficial. Even though there are few experimental studies, their results are largely inconclusive.

We reviewed existing experiments and related studies to gather a collection of potential effects of liveness and moderating factors. Based on this collection, we concluded that task complexity and prior experience addressing liveness are potentially essential factors neglected in previous experiments. To fill this gap, we devised and conducted a controlled experiment (N = 37) testing the hypothesis that task complexity moderates the effects of live introspection tools on participants’ debugging efficiency, given participants with prior experience with liveness.

Our results do not support the hypothesis that task complexity moderates the effect of live introspection tools. This non-significant moderation effect might result from the low number of participants, as the data suggests a moderation effect. The results also show that in our experiment setting, live introspection tools significantly reduced the time participants took to debug the tasks.

For researchers interested in the effects of liveness, our findings suggest that studies on liveness should make conscious choices about task complexity and participants’ prior experience with liveness. For designers of programming environments, the results of our experiment are a step toward understanding when and how programming tools should be live to become more helpful to programmers.