Robust Methods in Regression Analysis – Theory and Application

Table of Contents

1. Introduction

2. The Classical Linear Ordinary Least Squares Regression

2.1. Introduction to OLS

2.2. Properties of the Least Squares Estimates

2.3. Problems of OLS

3. Outliers and OLS

3.1. Outlier definition and common error sources

3.2. Outlier in Regression Analysis and their influence on OLS results

4. The concept of Robustness

4.1. Introduction to Robustness

4.2. Qualitative Robustness

4.3. Infinitesimal Robustness

4.4. Quantitative Robustness

4.5. Robust Estimates

4.6. On asymptotic Results

5. Some measures of location and scale – with regard to their robustness properties

5.1. Introduction

5.2. Measures of location

5.2.1. A Definition

5.2.2. The Arithmetic Mean

5.2.3. The Median

5.2.4. Trimmed mean(s)

5.2.5. Other measures of location

5.3. Measures of scale

5.3.1. A Definition

5.3.2. The Standard deviation

5.3.3. The Median Absolute Deviation (MAD)

5.3.4. The t-Quantile Range

5.3.5. Other scale estimates

5.4. Higher Dimensions

6. Robust Regression Techniques

6.1. An Introduction and Definition

6.2. M-Estimates

6.3. The Repeated Median

6.4. The Least Median of Squares Regression

6.5. The Least Trimmed Squares Regression

6.6. The Coakley – Hettmansperger Estimator

6.7. Reweighted Least Squares

6.8. The Multivariate Reweighted Least Squares Approach

6.8.1. The Hat Matrix

6.8.2. The Minimum Volume Ellipsoid Estimator

6.9. Other Regression Methods and Limitations

6.10. Conclusions on Robust Regression

7. Application to SAS and Simulation

7.1. Introduction to Robustness Application and Simulation purposes

7.2. The initial data set – The zero contamination case

7.3. Seemingly negligible contamination in X-direction

7.4. Seemingly negligible contamination in Y-direction

7.5. High Leverage contamination

7.6. Large overall contamination

8. Conclusions

Objectives and Topics

This thesis explores the vulnerability of the Ordinary Least Squares (OLS) regression method to outliers and evaluates alternative robust regression techniques. The primary objective is to technically examine how outliers affect OLS results and to introduce, discuss, and simulate more robust statistical estimators that provide reliable estimates in the presence of contaminated data.

• The theoretical foundations of Robustness (qualitative, infinitesimal, and quantitative).
• Evaluation of various measures of location and scale regarding their robustness properties.
• Detailed analysis of Robust Regression Techniques, including M-Estimates, Least Median of Squares, and Least Trimmed Squares.
• Implementation and comparative simulation using SAS to assess the performance of regression methods under different levels of outlier contamination.
• Practical recommendations for robust regression, emphasizing the Reweighted Least Squares (RLS) method.

Excerpt from the Book

Summary of Chapters

1. Introduction: Presents the motivation for robust statistics, highlighting the vulnerability of OLS regression to outliers and outlining the scope of the thesis.

2. The Classical Linear Ordinary Least Squares Regression: Recalls the OLS method, its necessary assumptions, and its key properties like the Best Linear Unbiased Estimator (BLUE) status.

3. Outliers and OLS: Defines outliers and investigates their impact on regression results, specifically distinguishing between leverage points and vertical outliers.

4. The concept of Robustness: Introduces the three fundamental pillars of robustness theory: qualitative, infinitesimal, and quantitative robustness, which serve as evaluation tools for estimators.

5. Some measures of location and scale – with regard to their robustness properties: Analyzes traditional and robust estimators for center and dispersion, serving as a basis for understanding robust regression.

6. Robust Regression Techniques: Discusses detailed robust alternatives to OLS, covering M-Estimates, Repeated Median, Least Median of Squares, Least Trimmed Squares, and Reweighted Least Squares.

7. Application to SAS and Simulation: Provides a simulation-based comparison of OLS, M-estimation, and high-breakdown techniques under various contamination scenarios.

8. Conclusions: Synthesizes the results and recommends the Reweighted Least Squares method as a robust and efficient approach for practical regression analysis.

Keywords

Regression Analysis, Ordinary Least Squares, Robustness, Outliers, Leverage Points, Breakdown Point, Influence Function, M-Estimates, Least Median of Squares, Least Trimmed Squares, Reweighted Least Squares, Asymptotic Efficiency, Simulation, SAS, Statistical Inference.

Frequently Asked Questions

What is the core focus of this thesis?

The work primarily addresses the sensitivity of standard regression models to outlier contamination and identifies robust alternative methods that remain reliable when data violates classical assumptions.

What are the central themes covered?

The thesis explores the theoretical concepts of robustness, analyzes various estimators for location, scale, and regression, and performs extensive simulation studies to compare their performance.

What is the primary research goal?

The main goal is to evaluate techniques that maintain high statistical performance in the presence of outliers and to provide actionable recommendations for robust regression analysis.

Which statistical methods are discussed?

The thesis covers OLS, M-Estimates, the Repeated Median, Least Median of Squares (LMS), Least Trimmed Squares (LTS), and Reweighted Least Squares (RLS).

What does the main part of the work involve?

The main section evaluates specific robust regression estimators through their robustness properties, computational feasibility, and efficiency, culminating in simulation-based benchmarking.

Which keywords characterize this work?

Key terms include Robustness, Outliers, Breakdown Point, Regression Analysis, Least Trimmed Squares, and Statistical Simulation.

How does the author define a "bad" versus a "good" leverage point?

A leverage point is a data point with an x-value far from the bulk of data. A leverage point is "good" if it fits the linear trend of the majority of data, thereby reducing estimator variance; it is "bad" if it deviates from that trend and exerts undue influence on the regression line.

Why is the "masking effect" important in outlier detection?

The masking effect occurs when outliers prevent the detection of other outliers, rendering standard diagnostics like the Hat Matrix unreliable. This is why the thesis advocates for initial robust estimation to correctly identify these influential points.