A Graph Theoretical Approach to Pruning Deep Neural Networks

Inhaltsverzeichnis (Table of Contents)

• 1 Introduction
• 1.1 Motivation
• 1.2 Proposal
• 1.3 Contributions
• 1.4 Thesis Structure
• 2 Background
• 2.1 Multilayer Perceptrons
• 2.2 Inference Optimisation
• 2.3 Weighted Page Rank
• 2.4 Related Work
• 3 The MLP-Rank Method
• 3.1 Graph Representation
• 3.2 Pruning Measure
• 3.2.1 Structural Adaptations
• 3.2.2 Theoretical Properties
• 3.2.3 Modified Page Rank
• 4 Experiments
• 4.1 Experiment Setup
• 4.1.1 Models and Datasets
• 4.1.2 Implementation Details
• 4.2 Sparsification Performance
• 4.2.1 Impact on Accuracy
• 4.2.2 Inference Speedup
• 4.3 Scoring Analysis
• 4.3.1 Realism of Theoretical Assumptions
• 4.3.2 Empirical Comparison of Importance Scores
• 5 Discussion
• 5.1 Theory and Practice
• 5.2 Outlook

Zielsetzung und Themenschwerpunkte (Objectives and Key Themes)

This bachelor's thesis aims to develop and evaluate a novel method for pruning deep neural networks using graph theory. The research explores the application of weighted PageRank to identify and remove less important connections within the network, leading to a more efficient and potentially faster inference process. The method is tested on various datasets and model architectures to assess its impact on accuracy and inference speed.

• Graph-based representation of neural networks
• Application of weighted PageRank for node importance scoring
• Development of a novel pruning algorithm (MLP-Rank)
• Evaluation of the algorithm's impact on accuracy and inference speed
• Comparison of theoretical and empirical performance

Zusammenfassung der Kapitel (Chapter Summaries)

1 Introduction: This introductory chapter sets the stage for the thesis by establishing the motivation behind researching efficient deep neural network pruning techniques. It highlights the challenges associated with the computational cost of deep learning models and introduces the proposed MLP-Rank method as a solution. The chapter outlines the contributions of the research and provides a roadmap to the thesis structure, outlining the flow of information across the subsequent chapters. The motivation is strongly tied to the need for optimized inference, emphasizing practical applications in resource-constrained environments. 2 Background: This chapter lays the groundwork for the thesis by providing a comprehensive overview of the fundamental concepts and related work. It introduces multilayer perceptrons (MLPs), focusing on their architecture and computational properties. The chapter then delves into inference optimization techniques and provides a detailed explanation of weighted PageRank, a crucial algorithm used in the proposed method. Finally, it discusses existing literature on network pruning, contextualizing the current research within the broader field. This background is essential for understanding the technical aspects of the thesis. 3 The MLP-Rank Method: This chapter presents the core contribution of the thesis – the MLP-Rank method. It details the process of representing an MLP as a graph, and explains how weighted PageRank is employed to assign importance scores to the network's connections. The chapter thoroughly analyzes the theoretical properties of the method and describes several structural adaptations made to enhance performance and effectiveness. A crucial aspect is the modification of the standard PageRank algorithm to better suit the specific characteristics of neural network graphs. This chapter is the technical heart of the thesis. 4 Experiments: This chapter meticulously describes the experimental setup used to evaluate the MLP-Rank method. It details the chosen models (MLPs with varying layers) and datasets (MNIST, Fashion-MNIST, CIFAR-10), and provides specific information on the implementation details. The experiments assess the method's performance in terms of sparsity, accuracy, and inference speedup across different pruning levels. A key aspect is the analysis of how well the theoretical assumptions hold up in practice, comparing the predicted speedup to the actual improvement observed. The results presented in this chapter are critical for validating the proposed method. 5 Discussion: This chapter offers a critical analysis of the results presented in Chapter 4. It contrasts the theoretical predictions of the MLP-Rank method with the empirical findings, analyzing the discrepancies and their potential causes. A significant part of this chapter is dedicated to discussing the practical implications of the research and outlining areas for future work. The overall effectiveness and limitations of the proposed method are thoughtfully discussed.

Schlüsselwörter (Keywords)

Deep Neural Networks, Network Pruning, Graph Theory, Weighted PageRank, Inference Optimization, Sparsity, Accuracy, Speedup, MLP-Rank, Model Compression.

Häufig gestellte Fragen

What is the "Language Preview" document about?

The document is a language preview for a bachelor's thesis. It includes the table of contents, objectives and key themes, chapter summaries, and keywords.

What is the central topic of the bachelor's thesis?

The thesis focuses on developing and evaluating a novel method for pruning deep neural networks using graph theory, specifically applying weighted PageRank to identify and remove less important connections to improve efficiency and inference speed.

What is the MLP-Rank method?

The MLP-Rank method is a pruning algorithm developed in the thesis. It represents a multilayer perceptron (MLP) as a graph and uses weighted PageRank to assign importance scores to connections, allowing for targeted pruning of less important connections.

What are the main objectives of the thesis?

The main objectives are: developing a graph-based representation of neural networks, applying weighted PageRank for node importance scoring, creating the MLP-Rank pruning algorithm, evaluating the algorithm's impact on accuracy and inference speed, and comparing theoretical and empirical performance.

What are the key themes explored in the thesis?

Key themes include graph-based neural network representation, weighted PageRank application, pruning algorithm development, accuracy and inference speed evaluation, and theoretical vs. empirical performance comparison.

What datasets and models were used in the experiments?

The experiments used MLPs with varying layers and datasets like MNIST, Fashion-MNIST, and CIFAR-10.

What is the purpose of pruning neural networks?

Pruning aims to reduce the computational cost of deep learning models, leading to more efficient and potentially faster inference, especially in resource-constrained environments.

What does Chapter 1 (Introduction) cover?

Chapter 1 introduces the motivation for researching efficient deep neural network pruning techniques, presents the MLP-Rank method, outlines the research's contributions, and provides a roadmap of the thesis structure.

What does Chapter 2 (Background) cover?

Chapter 2 provides an overview of fundamental concepts and related work, including multilayer perceptrons (MLPs), inference optimization techniques, and a detailed explanation of weighted PageRank.

What does Chapter 3 (The MLP-Rank Method) cover?

Chapter 3 details the MLP-Rank method, explaining how MLPs are represented as graphs, how weighted PageRank is used to assign importance scores, and the structural adaptations made to enhance performance.

What does Chapter 4 (Experiments) cover?

Chapter 4 describes the experimental setup used to evaluate the MLP-Rank method, including models, datasets, implementation details, and the assessment of sparsity, accuracy, and inference speedup.

What does Chapter 5 (Discussion) cover?

Chapter 5 offers a critical analysis of the experimental results, contrasts theoretical predictions with empirical findings, discusses the practical implications of the research, and outlines areas for future work.

What are the keywords associated with the thesis?

The keywords include: Deep Neural Networks, Network Pruning, Graph Theory, Weighted PageRank, Inference Optimization, Sparsity, Accuracy, Speedup, MLP-Rank, Model Compression.