Software Measurement

Modern Software Measurement

A modern framework for software measurement focuses on delivering value and improving continuously. It is typically guided by principles such as:

• Identifying key entities within the software development ecosystem.
• Defining clear, actionable measurement goals aligned with business objectives.
• Understanding the organization’s process maturity and selecting appropriate metrics.

Identifying Key Entities for Measurement

In contemporary software engineering, we primarily measure three classes of entities:

• Processes: The workflows and activities used to create and maintain software (e.g., development lifecycle, CI/CD pipeline, testing process).
• Products: The software itself and associated artifacts (e.g., code, documentation, test suites, deployed application).
• Resources: The inputs required for the processes (e.g., development teams, tools, infrastructure, budget).

Each entity possesses both internal and external attributes:

• Internal attributes: Measurable characteristics inherent to the entity itself. Examples: code complexity, test coverage percentage, process cycle time.
• External attributes: Characteristics measurable only by observing the entity’s interaction with its environment. Examples: application response time under load, user satisfaction score (NPS), defect rate reported by users.

Relevant attributes for measurement include:

Processes

Processes are sequences of software-related activities designed to achieve a specific outcome.

Internal process attributes commonly measured today:

• Cycle Time: Time taken from starting work on an item (e.g., feature, bug fix) to its completion.
• Lead Time: Time taken from request of an item to its delivery.
• Deployment Frequency: How often new code is deployed to production.
• Change Fail Rate: Percentage of deployments causing a failure in production.

External process attributes include: Cost, predictability, effectiveness (delivering value), overall quality impact, and stability/resilience.

Products

Products encompass the delivered software and any artifact created during the lifecycle (code, design documents, test cases, user manuals).

Internal product attributes: Size (e.g., Lines of Code - LOC, Story Points), Code Complexity (e.g., Cyclomatic Complexity, Cognitive Complexity), Test Coverage, Code Churn, Security Vulnerabilities (Static Analysis results), Maintainability Index.

External product attributes: Reliability (e.g., Mean Time Between Failures - MTBF), Performance (e.g., response time, throughput), Usability (e.g., task success rate, usability heuristics compliance), Security (e.g., penetration test results), Efficiency, Portability, and Interoperability.

Resources

Resources are entities consumed or utilized by process activities. This includes personnel (teams), tools (IDE, CI/CD servers), infrastructure (cloud services, hardware), and methodologies.

Internal resource attributes: Team size, tool license cost, server CPU/memory capacity, team skill matrix.

External resource attributes: Team Velocity (in Agile contexts), tool effectiveness, infrastructure reliability (uptime), developer productivity/satisfaction.

Defining Relevant Measurement Goals

Measurements are only valuable if they provide insights that help understand and improve processes or products. Clearly defined goals are essential. Vague goals like ‘improve quality’ are less effective than specific goals like ‘reduce production bug rate by 15% within the next quarter’.

The Goal–Question–Metric (GQM) Approach

The GQM approach remains a highly relevant framework for defining meaningful metrics:

• Goal: Define the major objectives for the project, process, or organization (What do you want to achieve?).
• Question: Derive specific questions from each goal whose answers will indicate progress towards that goal (What do you need to know to track progress?).
• Metric: Identify the data and calculations needed to answer those questions (What specific data points will provide the answer?).

Example: Goal - Improve the efficiency of our bug fixing process. Question - How long does it typically take to resolve a critical bug? Metric - Measure the ‘Cycle Time’ for critical bugs (from ‘In Progress’ to ‘Resolved’ status).

Using GQM ensures that measurements are purposeful and directly linked to desired outcomes. Consider the audience (developers, managers, customers) when formulating goals and questions.

Basili & Rombach’s templates for structuring GQM definitions are still helpful:

• Purpose: Analyze the (process/product) to (characterize/evaluate/improve) it for the purpose of (understanding/managing/etc.). Example: Analyze the code review process to evaluate its effectiveness for the purpose of improving code quality.
• Perspective: Examine (cost/defects/efficiency) from the viewpoint of the (developer/manager/customer). Example: Examine deployment frequency from the viewpoint of the DevOps team.
• Environment: Describe the context (process factors, team experience, tools, constraints). Example: The development team uses Agile Scrum with 2-week sprints and relies heavily on automated testing.

Measurement and Process Improvement

Measurement is fundamental for:

• Understanding current performance (Processes and Products).
• Establishing baselines for comparison.
• Tracking progress towards goals and predicting future outcomes.
• Identifying areas needing improvement.

The sophistication of measurement often correlates with process maturity. While the original SEI Capability Maturity Model (CMM) described levels, its successor CMMI (Capability Maturity Model Integration) and other models (like Agile or DevOps maturity models) provide frameworks. The core principle remains: as processes become more defined and managed, measurement becomes more sophisticated and impactful.

Level 1 (Initial/Ad hoc): Processes are chaotic. Basic measurements (e.g., total bugs found) might be possible, but lack context. Focus is on establishing basic project tracking.

Level 2 (Managed/Repeatable): Basic project management practices are in place. Inputs (requirements size), resources (effort), constraints (schedule, cost), and outputs (system size) can be identified and tracked. Processes are repeatable but may not be standardized across the organization. Textual Description (replaces image): A simple flow shows Inputs entering a ‘Repeatable Process’ box, constrained by Resources, and producing Outputs.

Level 3 (Defined): Processes are standardized and documented across the organization. Intermediate activities and their inputs/outputs are understood. Measurement can track process adherence and effectiveness of specific activities (e.g., defect density found in code reviews vs. testing). Textual Description (replaces image): A process is broken down into sequential sub-activities (e.g., Activity A -> Activity B -> Activity C), each with defined inputs and outputs.

Level 4 (Quantitatively Managed): The organization uses statistical and quantitative methods to manage processes and quality. Metrics are used to control process performance and predict outcomes. Feedback loops inform adjustments. Textual Description (replaces image): Similar to Level 3, but includes feedback loops from later activities (e.g., Activity C) back to earlier ones (e.g., Activity A or B), indicating measurement-driven adjustments.

Level 5 (Optimizing): Focus is on continuous process improvement based on quantitative feedback and piloting innovative ideas. The process itself is adapted and optimized proactively. Measurement drives fundamental process changes.

At any given maturity level, an organization can typically collect meaningful measurements appropriate for that level and all preceding levels.

Matching Measurement to Maturity

Process maturity dictates what can be meaningfully measured. Combining GQM with an understanding of maturity helps select the most useful metrics.

• At lower maturity levels, requirements might be unstable, making requirement-based metrics difficult. Focus might be on basic project tracking (effort, schedule).
• As maturity increases (e.g., Level 2/3), requirements become more stable, allowing tracking of changes, types, and complexity. Process adherence can be measured.
• At higher levels (e.g., Level 4/5), sophisticated metrics tracking process performance variations, defect prediction, and optimization effectiveness become feasible.

GQM defines the why (the goal), while process maturity informs the what and how (what can be reliably measured and how). Together, they provide a practical context for establishing an effective software measurement program.