Empirical Investigations

Empirical Investigations in Software Engineering

Empirical investigation applies scientific methods to study software engineering phenomena, tools, techniques, or processes in a systematic way. It moves beyond anecdotal evidence or intuition to provide data-driven insights. Key principles include:

• Choosing an appropriate investigation technique.
• Clearly stating the research question or hypothesis.
• Carefully managing variables and context.
• Ensuring the investigation is meaningful and ethically conducted.

Choosing an Investigation Technique

Common techniques used in empirical software engineering include:

• Survey
• Case Study
• Formal Experiment

Survey

A survey collects data from a sample population, often through questionnaires or interviews, to understand characteristics, opinions, or trends regarding a past or present situation. It’s typically retrospective.

Example: Surveying developers across multiple companies to understand the adoption rates and perceived benefits of different code review practices. Control over variables is low; the focus is on capturing a snapshot of current practices or opinions.

Case Study

A case study is an in-depth investigation of a specific phenomenon, project, team, or organization within its real-life context. It often involves multiple data sources (interviews, observations, document analysis, metrics).

Example: Studying a single software team’s experience adopting Test-Driven Development (TDD), documenting the process, challenges encountered, perceived impact on code quality, and collected metrics (like defect rates before and after adoption). Control over variables is limited as the study occurs in a natural setting, but it provides rich contextual understanding.

Formal Experiment

An experiment is a controlled investigation designed to test a specific hypothesis by manipulating one or more independent variables and observing their effect on dependent variables, while controlling for other factors.

Example: Comparing the defect-finding effectiveness of two different code inspection techniques. Two groups of developers inspect the same code; one group uses Technique A (experimental group), the other uses Technique B (control group). The number of defects found (dependent variable) by each group is measured. The experimenter controls factors like the code being inspected, time limits, and participant experience levels (through random assignment or blocking).

Textual Description (replaces image concept): A diagram could illustrate the spectrum: Surveys offer broad reach but low control. Case studies offer depth and context with limited control. Experiments offer high control and causal inference but may have limited scope or artificial settings.

Choosing the right technique depends on:

• Timing: Is the event past (Survey, Case Study) or future (Case Study, Experiment)?
• Control: How much control do you have over variables? (Low control -> Survey/Case Study; High control -> Experiment).
• Replication: Can the situation be easily replicated? (Difficult replication favors Case Study/Survey).
• Cost & Resources: Experiments can be resource-intensive.
• Research Goal: Exploration (Case Study), Description (Survey), Causal Explanation (Experiment).

Stating the Hypothesis or Research Question

A clear statement of the goal is crucial. This is often framed as a testable hypothesis (a specific prediction) or a focused research question.

• Hypothesis Example: ‘Using Test-Driven Development leads to a lower pre-release defect density compared to traditional development approaches.’
• Research Question Example: ‘What are the main challenges developers face when using asynchronous programming models?‘

Managing Variables and Context

Understanding the factors (variables) that might influence the outcome is critical, especially for experiments where control is key.

• Independent Variables: Factors manipulated by the researcher (e.g., the development technique used, the tool provided).
• Dependent Variables: Outcomes measured to see the effect (e.g., defect count, task completion time, code complexity score).
• Confounding Variables: Factors other than the independent variable that could potentially affect the outcome and need to be controlled or accounted for (e.g., developer experience, task complexity).

In experiments, control is achieved through techniques like randomization, blocking (grouping similar subjects), or keeping conditions constant. In case studies and surveys, while control is limited, researchers must carefully document the context and potential influencing factors.

Making the Investigation Meaningful and Ethical

The goal is to generate useful knowledge. Investigations can serve several purposes:

Confirming or Refuting Theories/Wisdom

Testing claims about the effectiveness of tools, methods (e.g., Agile practices), or standards. Formal experiments provide stronger evidence for generalizability, while case studies/surveys can confirm applicability in specific contexts.

Exploring Relationships

Identifying correlations between different attributes (e.g., Does higher code complexity correlate with higher defect rates? Does team size impact communication overhead?). Surveys and case studies often suggest relationships, while experiments can test for causal links.

Evaluating Models and Tools

Assessing the accuracy of prediction models (e.g., effort estimation models) or the usability/effectiveness of development tools through empirical studies (e.g., A/B testing different UI designs, usability studies of an IDE feature).

Validating Measures

Empirically validating whether a proposed software metric actually reflects the intended attribute (as discussed in the ‘Software Measurement Validation’ section). This often involves correlating the new metric with existing measures or expert assessments.

Ethical Considerations

When involving human participants (developers, users), ethical principles must be followed: obtaining informed consent, ensuring confidentiality/anonymity, avoiding harm, and providing fair compensation or benefit where appropriate. Institutional Review Board (IRB) approval may be required.

Empirical investigation provides a rigorous foundation for understanding and improving software engineering practices, moving the field towards evidence-based decision making.