Least Squares Regressions with the Bootstrap

Table of Contents

1 Introduction

2 The model

2.1 The basic form

2.2 The disturbance term

3 Regression techniques

3.1 The method of least squares

3.1.1 Ordinary Least Squares

3.1.2 Generalized Least Squares

3.2 Alternative regression methods

4 Classical measures of performance

4.1 Bias

4.2 Variances

4.2.1 The variance of OLS

4.2.2 The variance of GLS

4.2.3 A remark on the variances

4.3 Confidence intervals

4.3.1 A remark on the critical values

4.3.2 A confidence interval for OLS

4.3.3 A confidence interval for GLS

4.4 Rate of convergence

5 The bootstrap

5.1 How does the bootstrap work?

5.2 When does the bootstrap work?

5.3 The non-parametric bootstrap

5.4 The parametric bootstrap

5.5 Why does the bootstrap work?

5.6 How many bootstrap repetitions?

5.7 On the size of each repetition

6 Regressions with the bootstrap

6.1 Case resampling

6.2 Residual resampling

6.3 Wild bootstrap

6.4 When to use which method?

7 Inference with the bootstrap

7.1 Variances with the bootstrap

7.2 Confidence intervals with the bootstrap

7.2.1 The percentile interval

7.2.2 The bootstrap-t interval

7.2.3 Other bootstrap intervals

7.3 Convergence with the bootstrap

8 Classical or bootstrap inference?

8.1 Which variance estimate?

8.2 Which confidence interval?

8.3 When the bootstrap fails

9 A practical test for the bootstrap

9.1 The datasets

9.1.1 The homoscedastic data

9.1.2 The heteroscedatic data

9.2 The simulations

9.2.1 Results simulation one

9.2.2 Results simulation two

9.3 Resumé of simulations

10 Concluding remarks

A Tables simulation one

A.1 Table of coefficient β̂1

A.2 Table of coefficient β̂2

B Tables simulation two

B.1 Table of coefficient β̂1

B.2 Table of coefficient β̂2

C Friedman test values

C.1 Test values simulation one

C.2 Test values simulation two

D Some further formulae

E Bibliography

Objectives and Topics

The primary objective of this work is to evaluate the performance of different bootstrap methods within the context of linear regression models. The paper investigates whether bootstrap inference provides meaningful improvements over classical statistical methods for measuring regression performance and constructing confidence intervals, particularly when model assumptions are violated.

• Theoretical foundations of linear regression models and least squares estimators.
• Introduction to various bootstrap resampling techniques (Case, Residual, and Wild bootstrap).
• Comparative analysis of bootstrap-based inference versus classical asymptotic inference.
• Practical simulation testing under both homoscedastic and heteroscedastic conditions.

Excerpt from the Book

Summary of Chapters

1 Introduction: This chapter introduces regression analysis as a core statistical challenge and outlines the motivation for using bootstrap methods as an alternative to classical asymptotic inference.

2 The model: This section defines the standard linear regression model in matrix form and establishes the fundamental assumptions, including the properties of the disturbance term.

3 Regression techniques: This chapter covers OLS and GLS estimation methods and briefly touches upon alternative regression techniques such as Least Absolute Deviations and Least Median of Squares.

4 Classical measures of performance: This chapter details classical metrics for regression evaluation, focusing on bias, variances of estimators, and the construction of confidence intervals.

5 The bootstrap: This chapter provides an introduction to the bootstrap, detailing its working mechanism, the difference between parametric and non-parametric approaches, and practical considerations for simulation.

6 Regressions with the bootstrap: This section describes three specific bootstrap resampling methods developed for regression settings: Case resampling, Residual resampling, and the Wild bootstrap.

7 Inference with the bootstrap: This chapter explains how to use bootstrap results to calculate variances and construct confidence intervals, including percentile and bootstrap-t intervals.

8 Classical or bootstrap inference?: This chapter compares classical and bootstrap inference, discussing the justification for additional computational effort and the conditions under which the bootstrap may fail.

9 A practical test for the bootstrap: This chapter presents a simulation study comparing classical standard errors with three different bootstrap methods under homoscedastic and heteroscedastic data conditions.

10 Concluding remarks: The final chapter summarizes the findings, noting that while the bootstrap is a powerful tool, the simulation results did not consistently demonstrate superior performance over classical methods in all scenarios.

Keywords

Bootstrap, Linear Regression, Ordinary Least Squares, Generalized Least Squares, Resampling, Case Resampling, Residual Resampling, Wild Bootstrap, Asymptotic Inference, Homoscedasticity, Heteroscedasticity, Variance Estimation, Confidence Intervals, Simulation, Statistical Inference.

Frequently Asked Questions

What is the core focus of this diploma thesis?

The work focuses on analyzing the performance of different bootstrap resampling methods for statistical inference in the context of linear least squares regression models.

What are the primary regression techniques discussed?

The paper primarily examines Ordinary Least Squares (OLS) and Generalized Least Squares (GLS), while also discussing alternative methods like Least Absolute Deviations (LAD) and Least Median of Squares (LMS).

What is the main objective of the bootstrap in this study?

The bootstrap is employed as a tool for statistical inference, specifically to estimate variances of regression coefficients and construct confidence intervals when classical assumptions or asymptotic approximations may not yield satisfactory results.

Which bootstrap methods are analyzed in detail?

The study analyzes three specific non-parametric bootstrap methods: Case resampling, Residual resampling, and the Wild bootstrap.

What methodology is used to evaluate the bootstrap?

The methodology combines theoretical analysis of bootstrap properties with a practical simulation study, using the Friedman test to compare the standard error estimates produced by different methods.

What are the key findings regarding the bootstrap?

The simulation results indicate that the bootstrap does not always outperform classical methods, and its performance is highly dependent on the nature of the data, such as homoscedasticity versus heteroscedasticity.

Why is the "Wild Bootstrap" considered a hybrid method?

It is considered a hybrid because it utilizes residuals like residual resampling, but it introduces randomness through a multiplicative factor to better handle non-iid errors, similar to case resampling.

Does the bootstrap always work as expected?

No, the paper explicitly notes circumstances where the bootstrap fails, such as with autocorrelated time series data, when the parameter-to-sample-size ratio is too high, or when the incorrect bootstrap method is matched with the data structure.