A language-parametric test coverage framework for executable domain-specific languages

Abstract

Test coverage is an effective technique to measure test case quality and to enable fault localization. However, for Executable Domain-Specific Languages (xDSLs), coverage metrics and associated tools are currently manually defined for each xDSL resulting in costly, error-prone, and non-reusable work. To address this problem, we propose a novel language-parametric test coverage framework for xDSLs. We define two coverage metrics adapted to xDSLs: model element and branch coverage. For performing coverage measurements, we propose a generic technique which can be used out-of-the-box by domain experts using any xDSL to define, execute, and test models. In addition, the coverage of model elements and branches can be parameterized for a given xDSL through the definition of coverage rules using a dedicated language. We showcase two test coverage applications for xDSLs: measuring the quality of test suites for executable models and localizing the models’ faults using Spectrum-Based Fault Localization techniques. We evaluate our approach using four different xDSLs. Results show that (i) we can generate meaningful coverage measurements for all investigated models, (ii) the provided coverage rule language enables framework parameterization for all xDSLs, and (iii) the computed coverage measurements are useful in identifying defects of the models.