Can static type systems speed up programming? An experimental evaluation of static and dynamic type systems

Table of Contents

1. Introduction

2. Motivation & Background

2.1 Motivation

2.2 Maintenance and Debugging

2.2.1 Maintenance in a Nutshell

2.2.2 Debugging in a Nutshell

2.3 Documentation and APIs

2.3.1 Documentation of Software Systems

2.3.2 APIs and Application of their Design Principles in General Programming

2.4 Type Systems

2.5 Empirical Research in Software Engineering

2.5.1 On Empirical Research

2.5.2 Controlled Experiments

2.5.3 Current State of Empirical Research in Software Engineering

3. Related Work

3.1 Gannon (1977)

3.2 Prechelt and Tichy (1998)

3.3 Daly, Sazawal and Foster (2009)

3.4 Hanenberg (2010)

3.5 Steinberg, Mayer, Stuchlik and Hanenberg - A running Experiment series

3.5.1 Steinberg (2011)

3.5.2 Mayer (2011)

3.5.3 Stuchlik and Hanenberg (2011)

4. The Experiment

4.1 The Research Question

4.2 Experiment Overview

4.2.1 Initial Considerations

4.2.2 Further Considerations: Studies on Using Students as Subjects

4.2.3 Design of the Experiment

4.3 Questionnaire

4.4 Hard- and Software Environment

4.4.1 Environment

4.4.2 Programming Languages

4.5 Workspace Applications and Tasks

4.5.1 The Java Application - A Labyrinth Game

4.5.2 The Groovy Application - A simple Mail Viewer

4.5.3 Important Changes made to both Parts

4.5.4 The Tasks

4.6 Experiment Implementation

5. Threats to Validity

5.1 Internal Validity

5.2 External Validity

6. Analysis and Results

6.1 General Descriptive Statistics

6.2 Statistical Tests and Analysis

6.2.1 Within-Subject Analysis on the complete data

6.2.2 Analysis for residual effects between the two Participant Groups

6.2.3 Within-Subject Analysis on the two Participant Groups

6.2.4 Exploratory Analysis of the Results based on Participants’ Performance

6.2.5 Hypotheses and Task based Analysis

7. Summary and Discussion

7.1 Final Remarks

7.2 Result Summary

7.3 Discussion

8. Conclusion

Research Objectives and Core Themes

The primary objective of this thesis is to empirically investigate whether static type systems enhance developer performance during software maintenance tasks, specifically when working with undocumented APIs. The study aims to clarify the ongoing debate regarding the practical advantages of static versus dynamic type systems by conducting a controlled experiment, thereby narrowing the gap between theoretical speculation and empirical evidence in software engineering.

• Impact of static versus dynamic type systems on developer performance.
• Role of API documentation and variable naming in maintenance tasks.
• Influence of task complexity (number of types) on development time.
• Distinction between semantic errors and latent type errors in programming.
• Methodological rigor in empirical software engineering experiments.

Excerpt from the Book

Summary of Chapters

1. Introduction: Presents the motivation for the study, highlighting the controversial nature of static versus dynamic type systems and the scarcity of empirical research on their impact on software maintenance.

2. Motivation & Background: Discusses core concepts including software maintenance, debugging, documentation, APIs, and the theoretical underpinnings of different type systems, alongside an overview of empirical research methods.

3. Related Work: Reviews existing literature and previous empirical studies regarding type systems, emphasizing the need for rigorous experimental designs in software engineering.

4. The Experiment: Details the design, research questions, hypotheses, software environment, and specific task categories utilized in the controlled experiment.

5. Threats to Validity: Analyzes potential limitations of the experiment, such as learning effects, participant variance, and the use of students as subjects, to ensure critical interpretation of the results.

6. Analysis and Results: Provides a comprehensive statistical evaluation of the gathered data, including descriptive statistics, within-subject analyses, and hypothesis testing.

7. Summary and Discussion: Synthesizes the main findings, discusses the validity of the experimental approach, and reflects on the implications of the results for software engineering practices.

8. Conclusion: Summarizes the study’s contributions, concluding that the results support the hypothesis that static type systems can positively impact developer performance in specific maintenance scenarios.

Keywords

Static type systems, dynamic type systems, software maintenance, developer performance, controlled experiment, empirical software engineering, undocumented API, debugging, latent type errors, semantic errors, programming languages, Java, Groovy, task complexity, experimental design.

Frequently Asked Questions

What is the core focus of this empirical study?

The study investigates the impact of static versus dynamic type systems on software developer performance, specifically during maintenance tasks involving undocumented APIs.

Which programming languages were compared in this research?

The experiment used Java to represent a statically typed language and Groovy to represent a dynamically typed language.

What is the central hypothesis of the experiment?

The primary hypothesis is that a static type system reduces development time when developers perform tasks using an undocumented API.

What research methodology was employed?

A controlled experiment with a repeated-measures design was conducted, involving 36 participants who performed various programming tasks.

What specific types of errors were the participants tasked with fixing?

Participants dealt with three task categories: type identification tasks, semantic errors, and latent type errors.

How is developer performance measured in this study?

Performance is primarily measured as the time required for a developer to successfully complete a given programming task.

How did the study address potential learning effects?

The study used a repeated-measures design and two participant groups (Java-first and Groovy-first) to detect and account for learning and carry-over effects.

What were the main conclusions regarding the hypotheses?

The main hypothesis regarding the benefit of static type systems was supported, while the additional hypotheses concerning error distance and semantic error fixing times were rejected due to inconclusive or contradictory data likely caused by learning effects.

Why are the participants primarily students?

The use of students is a common practice in software engineering research to manage resources and maintain experimental control, though the study discusses the implications of this choice on external validity.