Tools for monitoring software quality in information systems development and maintenance: five key challenges and a design proposal

As software grows in size and complexity, organizations increasingly apply tools to automatically assess software quality of information systems during development and maintenance. Software quality assessment tools (SWQAT) promise fast and actionable insights into the technical state of software through various quality characteristics, such as maintainability, reliability, or security. These tools have been used to support a wide variety of IT project management decisions related to system development, contract negotiations, project terminations, and even settling legal disputes between suppliers and clients. However, despite their rising importance, questions regarding how they function and how reliable they are to support decision-making have so far escaped scholarly attention. This paper conducts an evaluation of widely used SWQATs and analyzes how they rate the quality of software systems of varying sizes, functionalities, and programming languages. Our results reveal five key challenges for using SWQATs in IT projects. To address these challenges, we propose a design for tailorable SWQATs that allows for more conscious and prudent software quality assessments that better reflect the socio-technical aspect of software systems and the context-specific nature of software quality.