Detection of Sarcastic and Abusive Comments Containing Positive Words

Contents

1 Introduction
1.1 Background
1.2 Research Objective
1.3 Thesis Structure

2 Related Research
2.1 Tohoku University BERT Model
2.2 Classification and Detection of Sarcastic Expressions Using Pattern Matching
2.3 Detection of Sarcastic Expressions Containing Positive Words
2.4 Detection of Sarcastic Expressions Using Part-of-Speech-Based Phrases
2.5 Detection of Sarcastic Expressions Using BERT

3 Definition of Target Comments for Detection in This Study

4 Proposed Method
4.1 Fine-tuning
4.1.1 Binary Classification of Sentiment
4.1.2 Conditions for Considering as Detection Target
4.2 Pattern Matching
4.2.1 Keywords
4.2.2 Using Sentiment Analysis Results

5 Experiments
5.1 Dataset
5.2 Preprocessing
5.3 Results and Evaluation
5.3.1 Binary Classification of Sentiment
5.3.2 Conditions for Considering as Detection Target
5.3.3 Pattern Matching
5.3.4 Final Binary Classification
5.3.5 Ablation Study

6 Discussion
6.1 Dataset and Preprocessing
6.2 Factors for Accuracy Improvement Over Existing Methods . .
6.3 Relationship Between Threshold Values p, s and Detection Accuracy
6.4 False Detections by Pattern Matching
6.5 Future Improvements

7 Conclusion

References

List of Tables

1.1 Examples of comments incorrectly analyzed as positive by the Tohoku University BERT model

2.1 Part of the proposed syntactic patterns

2.2 List of expressions treated as exceptions

2.3 List of euphemistic/emphatic expressions used to narrow down detection target sentences

4.1 Examples of keywords selected for creating syntactic patterns

4.2 Number of comments used in the experiment

4.3 Settings of each argument in Early stopping

4.4 Comparison of sentiment analysis accuracy

4.5 Sentiment analysis results with the fine-tuned model

4.6 List of results for comments focused on in this study where binary classification was not performed correctly

4.7 List of results for comments focused on in this study where predicted label was 0 and probability of not being detection target was less than 0.90

4.8 List of results when condition that sentiment score value is greater than 0.90 is added

4.9 Predicted labels and actual labels in binary classification for comments corresponding to pattern matching

4.10 Comparison of classification accuracy between existing model and proposed model

4.11 Results of ablation study

LIST OF TABLES

6.1 Average values of probability of not being detection target and sentiment score

6.2 Pattern matching results when sentiment binary classification model was not used

List of Figures

4.1 Overview of the proposed model

Abstract

With the proliferation of social networking services (SNS), damage caused by sarcastic and abusive comments has been increasing, creating a demand for technologies that can automatically detect such comments. Among these, comments that use indirect expressions, such as sarcasm, without employing direct language are particularly difficult to detect. In this study, we focus on comments containing positive words as a form of indirect expression, and aim to construct a model capable of detecting sarcastic and abusive comments with higher accuracy than existing models by not only fine-tuning an existing BERT model but also incorporating several techniques such as pattern matching. As a result, we were able to surpass the existing model in all four evaluation metrics: Accuracy, Precision, Recall, and F1.

Chapter 1

Introduction

1.1 Background

In recent years, the proliferation of social networking services (SNS) has led to ongoing damage from sarcastic and abusive comments on anonymous sites, creating a significant social problem. Therefore, to address this issue, it is necessary to prevent inappropriate comments from being visible to users.

To achieve this, companies operating SNS platforms and other services where anyone can easily post comments are required to promptly detect and delete sarcastic and abusive comments. Indeed, several companies, including LINE Yahoo Corporation, have already deployed AI systems to automatically detect and remove inappropriate comments. However, such measures have yet to significantly reduce the damage[1]. One possible reason is that while sarcastic and abusive comments using direct language can be easily detected, comments using indirect expressions are difficult to identify. For example, when sentiment analysis was performed on several sarcastic and abusive comments using indirect expressions with the Tohoku University BERT model[2], many were classified as positive rather than negative.

Currently, extensive research is being conducted on detecting sarcastic and abusive comments, but there is limited research focusing on indirect expressions. Therefore, methods capable of detecting sarcastic and abusive comments using indirect expressions with high accuracy are needed.

1.2 Research Ob jective

Against this background, this study focuses on sarcastic and abusive comments containing positive words as one form of indirect expression, and proposes a model capable of detecting such comments with high accuracy. In recent natural language processing tasks, BERT (Bidirectional Encoder Representations from Transformers)[3], announced by Google in 2018, has achieved high performance; therefore, this study utilizes the pre-trained To- hoku University Japanese BERT model. Specifically, by fine-tuning the To- hoku University BERT model to enable more accurate sentiment analysis and combining it with pattern matching and other techniques, we aim to construct a model capable of correctly performing binary classification when only the comment is provided.

As mentioned in the background, sarcastic and abusive comments containing positive words are often incorrectly classified as positive by sentiment analysis; therefore, this study primarily focuses on such comments. Table 1.1 shows examples of comments that are actually negative but were incorrectly analyzed as positive when sentiment analysis was performed using the To- hoku University BERT model.

Table 1.1: Examples of comments incorrectly analyzed as positive by the

Tohoku University BERT model

Illustrations are not included in the reading sample

1.3 Thesis Structure

This thesis consists of Chapters 1 through 7. The specific structure is as follows:

• Chapter 2: Introduction of related research
• Chapter 3: Definition of target comments for detection in this study
• Chapter 4: Explanation of the proposed method in this study
• Chapter 5: Experiments using existing and proposed methods and their results
• Chapter 6: Discussion based on experimental results
• Chapter 7: Summary of this study and future prospects

Chapter 2

Related Research

2.1 Tohoku University BERT Model

The Tohoku University BERT model is a pre-trained Japanese BERT model released by the Natural Language Processing Research Group at To- hoku University. Japanese Wikipedia was used as the corpus for training. In tokenization, text is first divided into words using MeCab (ipadic)[4], and then words are divided into tokens using WordPiece[5]. This model is also available as a pre-trained model in Transformers, a natural language processing library by HuggingFace[6].

In this study, we construct a model capable of correctly performing sentiment analysis and binary classification of whether a comment is a detection target when only the text is provided, by fine-tuning the pre-trained Tohoku University BERT model.

2.2 Classification and Detection of Sarcastic Expressions Using Pattern Matching

Sarcastic expressions are frequently used as a representative example of indirect expressions. Isono et al. collected approximately 14,600 Japanese text data from Rakuten Travel review data and web bulletin boards, manually labeled them for sarcastic expression inclusion, and constructed a corpus[7].

By analyzing the corpus, they designed syntactic patterns and proposed a model that classifies sarcastic expressions into eight categories through pattern matching: question, speculation, resignation, inappropriateness, exaggeration, surprise, description, and contrast. Reasoning that context before and after is important in judging whether an expression is sarcastic, the pattern matching in this research focused on sentences before and after those matching syntactic patterns and considered whether negative terms listed in the Japanese Sentiment Polarity Dictionary[8] were included in the preceding and following sentences. Table 2.1 shows excerpts from the proposed syntactic patterns classified as exaggeration that contain positive words.

Table 2.1: Part of the proposed syntactic patterns

Illustrations are not included in the reading sample

2.3 Detection of Sarcastic Expressions Containing Positive Words

Hiai et al. focused on expressions containing positive words among sarcastic expressions and performed detection of corresponding sentences through two stages[9]. However, cases where positive words were used in forms of wishes, regrets, or negations were treated as exceptions. Table 2.2 shows a list of expressions treated as exceptions. In performing detection, they first obtained 70 review texts that were detection targets from Rakuten Market review data. Then, they analyzed how positive expressions represent negative meanings in each acquired sentence and classified them into eight classes. Examples of classes included sentences that directly paraphrase negative evaluations of the target positively, and sentences that show negative evaluation through contrast by positively evaluating other similar targets. After classification into eight classes, in the first stage, sentences that were candidates for detection were narrowed down using the overall polarity of sentences and syntactic patterns. Next, in the second stage, only sentences containing any of the euphemistic/emphatic expressions or symbols listed in Table 2.3 among the remaining candidate sentences were ultimately extracted as detection targets.

Table 2.2: List of expressions treated as exceptions If it were, if it had been, if it were, if it had been, even if, even if it had been, it would have been better if, it would have been better if, well okay, will accept

Table 2.3: List of euphemistic/emphatic expressions used to narrow down detection target sentences

Illustrations are not included in the reading sample

2.4 Detection of Sarcastic Expressions Using Part-of-Speech-Based Phrases

Suzuki proposed a sarcasm detection method using phrases formed by combinations of parts of speech such as nouns and adjectives[10]. The proposed phrases were formed from situation phrases representing actions and sentiment phrases representing emotions, each divided into two types: positive phrases and negative phrases. The emotion of the phrase was determined by whether there were more positive or negative phrases, and when the emotions of situation phrases and sentiment phrases differed, it was judged as a sentence containing sarcastic expression. When the numbers of positive and negative phrases were equal, the phrase emotion was determined by the tf-idf method.

2.5 Detection of Sarcastic Expressions Using BERT

Since sarcastic expressions exist not only in Japanese but in various other languages, research on detecting sarcastic expressions is conducted for each language. As an example, Yazhou et al. performed detection of sarcastic expressions in Chinese and English using BERT[11]. In this research, for each comment prepared as a dataset, the position of the comment writer was manually selected by annotators from among agreeing, opposing, and neutral, and a graph attention network was formed with a node representing that position at the center, connected to nodes of word embedding vectors for each word contained in the comment. They proposed a method that learns sentence distributed representations through BERT and stance representations through the graph attention network, concatenates them, and passes them to a softmax layer to detect sarcastic expressions. As a result, they were able to detect sarcastic expressions with higher accuracy than baseline methods such as SVM and CNN in three evaluation metrics—Accuracy, Precision, and F1—for both Chinese and English datasets.

Chapter 3

Definition of Target Comments for Detection in This Study

The comments that are detection targets in this study are, as stated in the title, ”sarcastic and abusive comments containing positive words,” which includes the condition that ”they appear to be positive comments at first glance but are actually negative comments”—that is, the condition that they contain sarcastic expressions. Therefore, cases where positive words are used in negative form are exceptionally not treated as detection targets. For example, sentences like ”it’s not interesting” or ”you can’t say it’s amazing” contain positive words partially but are used in negative form, so they are treated as exceptions. Also, words like ”lol” (^) or ”w” that are often used at the end of sentences can be considered positive words, but since such words are widely used as part of sarcastic expressions, they are not considered positive words in this study.

Chapter 4

Proposed Method

In this study, we propose a model with higher detection accuracy for sarcastic and abusive comments containing positive words than existing BERT models by fine-tuning the pre-trained Tohoku University BERT model and introducing pattern matching. The specific overview of the proposed model is shown in Figure 4.1. Note that the final predicted label means predicting that it is not a detection target when it is 0, and predicting that it is a detection target when it is 1.

Illustrations are not included in the reading sample

Figure 4.1: Overview of the proposed model

4.1 Fine-tuning

4.1.1 Binary Classification of Sentiment

In this study, considering the characteristics of sarcastic expressions, we focus on comments like those listed in Table 1.1 that are actually negative but are incorrectly analyzed as positive. However, since the purpose of this study is to propose a model that correctly classifies whether a comment contains positive words and is intended for sarcasm or abuse when only the comment is given, we do not intend to manually assign the actual sentiment (positive, neutral, or negative) to each comment every time. Therefore, it is necessary to be able to automatically predict and assign the actual sentiment to comments.

Therefore, rather than simply performing binary classification of whether a comment contains positive words and is intended for sarcasm or abuse when given a comment, we first fine-tune the Tohoku University BERT model to enable highly accurate analysis of actual sentiment. At that time, since there were extremely few comments whose actual sentiment was positive compared to comments whose sentiment was neutral or negative in the dataset used in this study, we decided to perform binary classification of positive or neu- tral/negative rather than three-class classification. Therefore, for all comments included in the dataset, label 0 is manually assigned if the actual sentiment is positive or neutral, and label 1 is assigned if it is negative.

4.1.2 Conditions for Considering as Detection Target

The simplest condition for considering as a detection target is that the predicted label becomes 1 when binary classification is performed on whether a comment is a detection target in this study by fine-tuning the Tohoku University BERT model. When this happens, the final predicted label is set to 1 at that point and it is considered a detection target. From this point on, we describe conditions for considering as a detection target when the predicted label of binary classification is 0.

As already mentioned, this study focuses on comments that are actually negative but are incorrectly analyzed as positive by existing models. That is, we focus on ”comments that are analyzed as positive by the Tohoku University BERT model but are analyzed as negative by a fine-tuned model” using a model that has become capable of more accurate sentiment analysis than the Tohoku University BERT model for comments containing sarcastic expressions, and consider comments that meet this criterion as candidates for detection targets in this study.

Comments that meet the above conditions are likely to be detection targets in this study, but if all matching comments are considered detection targets, detection accuracy will conversely decrease. Therefore, we add two conditions to further narrow down candidates for sarcastic and abusive comments containing positive words.

The first condition is that the probability of not being a detection target is smaller than threshold value p as a result of binary classification on whether it is a detection target. The reason for adding this condition is that we considered that if the probability of not being a detection target is low, the possibility that the predicted label is incorrect becomes higher. In binary classification, classification scores for both label 0 and label 1 are output for each comment, and the one with the larger value becomes the predicted label, so classification scores can be changed to probability values by inputting the two classification scores into the softmax function. The softmax function is expressed by the following formula. n represents the number of types to classify, and x i represents the value of each original classification score. Since binary classification is performed here, in formula 4.1, the value of i is i =1 , 2 and the value of n is n =2.

Illustrations are not included in the reading sample

Next, as the second condition, we use sentiment scores. With the Tohoku University BERT model, when a sentence is input, not only can we determine whether it corresponds to positive, neutral, or negative, but we can also calculate a sentiment score, which represents the degree to which the given sentence is biased toward that sentiment, in the range of 0 ~ 1. Comments that are detection targets in this study have the premise of containing positive words, but when sarcastic expressions are used, positive words are considered to be more emphasized; therefore, we add the condition of designating as detection targets only when the sentiment score value is greater than threshold value s.

By considering comments that satisfy both of these two conditions as detection targets—that is, by changing the predicted label from 0 to 1—we improve accuracy compared to when simply fine-tuning the Tohoku University BERT model and performing binary classification.

4.2 Pattern Matching

A method frequently used in detecting sarcastic expressions is called pattern matching. Pattern matching in sentence detection or classification is a method of creating syntactic patterns by considering what patterns exist as characteristics of sentences to be detected, as shown in Table 2.1, and designating sentences that match those syntactic patterns as detection targets. By applying pattern matching to comments predicted not to be detection targets in binary classification on whether they are detection targets, we aim to reduce the number of comments erroneously predicted not to be detection targets.

A problem with pattern matching is that false detections increase. When using pattern matching, all sentences corresponding to syntactic patterns become subject to detection or classification, so detection or classification is performed even for unintended sentences, and as a result, accuracy often worsens. However, as a countermeasure, if syntactic patterns are reduced too much, the number of false detections decreases, but accordingly the number of sentences that can be detected or classified also decreases, and as a result, the effectiveness of pattern matching is lost; therefore, when using pattern matching, it is necessary to appropriately set the number and conditions of syntactic patterns.

Therefore, to solve the problem that false detections particularly increase, this study proposes a method that adds several conditions to normal pattern matching.

4.2.1 Keywords

Essential in creating syntactic patterns is keyword selection. In pattern matching, it is necessary to select keywords useful for accurate sentence detection or classification and create syntactic patterns that include several such keywords. In this study, we established keywords separately for within sentences and at sentence ends. This is because we considered that comments containing sarcastic expressions have characteristics not only in words within sentences but also in words at sentence ends. Table 4.1 shows excerpts of examples of keywords established in this study.

Table 4.1: Examples of keywords selected for creating syntactic patterns

Illustrations are not included in the reading sample

For each keyword, when there are various ways of writing such as kanji, hiragana, and katakana, those are also included in the keywords. For example, for ”gyaku ni” (on the contrary), ”^^< }C” is also included as a keyword, and for ”warota” (LOL), ”7n^” is also a keyword. Keywords also include not only words but also emoticons, emoji, and symbols.

Comments that contain any selected keyword in both within sentences and at sentence ends are considered to match the syntactic pattern.

4.2.2 Using Sentiment Analysis Results

However, this condition alone does not solve the problem of many false detections in pattern matching. Therefore, we use sentiment analysis results.

First, for comments that match the condition (syntactic pattern) mentioned above, we extract the part after the keyword within the sentence. Then, we perform sentiment analysis on the extracted part using the Tohoku University BERT model and designate only comments whose result is positive as detection targets.

To further reduce false detections, for comments that satisfy these conditions, we perform sentiment analysis using the model fine-tuned in Section 4.1.1 to enable more accurate binary classification of sentiment, and consider comments whose result is negative as final detection targets. That is, we change the predicted label from 0 to 1.

Chapter 5

Experiments

5.1 Dataset

The comments used in the experiment were obtained from the comment section of one YouTube video selected from a website[12] that lists rankings of all Japanese YouTube videos by number of dislikes. Note that only comments on the video were obtained; comments replying to other users were not obtained. Also, if all comments were randomly obtained, the number of comments that are detection targets would be much smaller relative to comments that are not detection targets, so the comment ratio was adjusted to maintain a certain proportion.

The total number of comments obtained in this way is shown in the following table.

Table 5.1: Number of comments used in the experiment

Illustrations are not included in the reading sample

During the experiment, this dataset was randomly shuffled and then divided into training data 60%, validation data 20%, and test data 20%.

5.2 Preprocessing

Following dataset collection, we performed preprocessing on each collected comment to enable more accurate binary classification. The preprocessing actually performed is shown below.

• Removal of half-width symbols
• Removal of URLs
• Removal of spaces
• Unification of numbers to 0
• Unification of alphabetic characters to lowercase

In sentence preprocessing, emoticons and emoji are often removed, but since they are necessary for pattern matching, they were left without removal in this study.

5.3 Results and Evaluation

5.3.1 Binary Classification of Sentiment

Fine-tuning was performed on the Tohoku University BERT model using the method described in Section 4.1.1. At that time, the learning rate was set to 1 x 10 _ 6 and the batch size was set to 64. Also, the number of epochs was not specified; instead, Early stopping was introduced. The settings of each argument in Early stopping are shown in Table 5.2.

Table 5.2: Settings of each argument in Early stopping

Illustrations are not included in the reading sample

As shown in Table 5.2, settings were made so that learning stops if the validation loss value does not improve for 20 epochs, using the validation loss value as the criterion. Also, mode was specified to issue convergence determination when converged to the lower limit.

Under the above settings, we compared the accuracy of sentiment analysis between the Tohoku University BERT model and the fine-tuned model using all comments included in the dataset. As evaluation metrics for comparison, we used four metrics: Accuracy, Precision, Recall, and F1. The results of the accuracy comparison are shown in the following table.

Table 5.3: Comparison of sentiment analysis accuracy

Illustrations are not included in the reading sample

Looking at Table 5.3, it can be seen that the fine-tuned model is superior to the Tohoku University BERT model in all evaluation metrics. Here, to confirm whether the fine-tuned model is useful for detecting sarcastic and abusive comments containing positive words, we performed sentiment analysis with the fine-tuned model on the comments listed in Table 1.1 whose actual sentiment is negative but were incorrectly analyzed as positive by the Tohoku University BERT model. The results are shown in Table 5.4.

Table 5.4: Sentiment analysis results with the fine-tuned model

Illustrations are not included in the reading sample

From these results, it was confirmed that for comments containing sarcastic expressions that were incorrectly analyzed as positive by the Tohoku University BERT model, the fine-tuned model could more accurately analyze them as negative, confirming that this model can be utilized for detecting sarcastic and abusive comments containing positive words.

5.3.2 Conditions for Considering as Detection Target

Next, by fine-tuning the Tohoku University BERT model, binary classification was performed on all comments included in the dataset regarding whether they contain positive words and are intended for sarcasm or abuse. Without adding the conditions described in Section 4.1.2, for all comments that were analyzed as positive by the Tohoku University BERT model, negative by the fine-tuned model in Section 5.3.1, and had different predicted and actual labels in binary classification on whether they are detection targets, the probability of being label 0 (not a detection target), predicted label, and actual label are shown in the following table.

Table 5.5: List of results for comments focused on in this study where binary classification was not performed correctly

Illustrations are not included in the reading sample

From this table, it was found that when predicted and actual labels differ, most cases have predicted label 0 and actual label 1.

Next, setting the threshold value p defined in Section 4.1.2 to p = 0 . 90, for all comments that satisfy the condition that the predicted label is 0 as a result of binary classification and the probability of not being a detection target is less than 0.90, the probability of not being a detection target, predicted label, actual label, and sentiment score value calculated using the Tohoku University BERT model are shown in the following table.

Table 5.6: List of results for comments focused on in this study where predicted label was 0 and probability of not being detection target was less than

Illustrations are not included in the reading sample

From Table 5.6, it was found that simply changing the predicted label to 1 for all comments focused on in this study where the predicted label in binary classification is 0 and the probability of not being a detection target is less than 0.90 would increase values like Recall but increase false detections and the Accuracy value would hardly change, making it not a good strategy.

Then, setting the threshold value s defined in Section 4.1.2 to s = 0 . 90, and adding the condition described as the second condition that only when the sentiment score value calculated using the Tohoku University BERT model is greater than 0.90 should it be designated as a detection target, when comments satisfying these conditions are tabulated similarly to before, the result is as follows.

Table 5.7: List of results when condition that sentiment score value is greater than 0.90 is added

Illustrations are not included in the reading sample

Therefore, from Table 5.7, since comments that satisfy all the conditions described in the proposed method all match with predicted label 0 and actual label 1 as binary classification results, by changing the predicted label to 1 for these comments, we achieved higher accuracy than when simply fine-tuning the Tohoku University BERT model and performing binary classification.

5.3.3 Pattern Matching

Next, among comments predicted not to be detection targets in binary classification on whether they are detection targets, we extracted comments that correspond to the pattern matching defined in Section 4.2, and when predicted labels and actual labels of those comments are arranged, the result is as follows.

Table 5.8: Predicted labels and actual labels in binary classification for comments corresponding to pattern matching

Illustrations are not included in the reading sample

Therefore, changing the predicted label to 1 for comments corresponding to the pattern matching defined in the proposed method increases false detections slightly, but introducing pattern matching improved accuracy.

5.3.4 Final Binary Classification

So far, we have seen changes in classification accuracy when fine-tuning to enable more accurate binary classification of sentiment, increasing detection target comments by focusing on the probability that the predicted label is 0 and sentiment score values, and introducing pattern matching that reduces false detections compared to conventional methods. Now we compare the final classification accuracy when these two methods are combined. The comparison target is a model that simply fine-tunes the existing Tohoku

University BERT model and performs binary classification.

As evaluation metrics, we used the same four metrics as when comparing sentiment analysis accuracy between the Tohoku University BERT model and the fine-tuned model in Section 5.3.1: Accuracy, Precision, Recall, and F1.

The results of comparing final classification accuracy are shown in Table 5.9.

Table 5.9: Comparison of classification accuracy between existing model and proposed model

Model Accuracy Precision Recall F1

Tohoku University BERT Model (Existing Model) 0.972 0.816 0.635 0.714

Proposed Model 0.976 0.818 0.714 0.763

From these results, the model proposed in this study surpassed the existing model in all four metrics. That is, it was found that the proposed model can perform binary classification on whether a comment contains positive words and is intended for sarcasm or abuse with higher accuracy than the existing model.

5.3.5 Ablation Study

From Table 5.9, it was shown that the proposed model has higher binary classification accuracy than the existing model, but to investigate how much each of the two methods using sentiment analysis results described mainly in Sections 4.1.1 and 4.1.2 and the method using pattern matching described in Section 4.2 contribute to performance improvement, when classification accuracy using only one of the two methods is added to Table 5.9, the result is as follows. For clarity, each method was numbered as follows.

Method using two sentiment analysis results ••• ((

Method using pattern matching ••• (2)

Illustrations are not included in the reading sample

From the ablation study results, since the method using two sentiment analysis results recorded equal or higher accuracy compared to the method using pattern matching in all metrics, it was found that the method using two sentiment analysis results contributes more to improving the accuracy of the proposed model. Also, particularly for the Precision value, there is a large difference between the two methods, and it was found that using only the method using pattern matching results in a lower Precision value than the existing model.

Chapter 6

Discussion

6.1 Dataset and Preprocessing

First, regarding the dataset, since it was obtained from one YouTube video, the number of detection target comments was small, and there were several comments with similar sentences among detection target comments, so it is considered that accuracy improved easily through fine-tuning. As can be seen from Table 5.1, despite the heavily skewed dataset ratio, high classification performance was achieved, so by adjusting the ratio a little more, improvement in overall classification performance and divergence in performance between existing and proposed models can be expected.

In preprocessing, removal of emoticons, emoji, and particles was not performed due to compatibility with pattern matching, but this can also be said to be a problem specific to using pattern matching; therefore, it is considered that improvements such as removing these characters before inputting to BERT when pattern matching is not applicable would have been necessary.

6.2 Factors for Accuracy Improvement Over Existing Methods

From the accuracy comparison results in Table 5.3, it was found that more accurate sentiment analysis becomes possible by fine-tuning the Tohoku University BERT model. Looking at the values of the four evaluation metrics listed in Table 5.3, it can be seen that there is particularly a difference in Recall values. From this, it can be said that the fine-tuned model analyzes comments that are actually negative as positive or neutral less often compared to the Tohoku University BERT model; therefore, the opening of the difference in Recall values made it possible to extract many comments like those listed in Table 5.4 that are important in this study, which is considered to have led to overall accuracy improvement.

Looking at Table 5.5, it can be seen that it is indeed difficult to correctly predict comments that are actually negative but incorrectly analyzed as positive as detection targets. On the other hand, it can also be seen that there are almost no errors when predicting such comments as detection targets. From this, it is considered that proposing a method that reduces mistakenly considering something not a detection target rather than reducing false detections was a good choice.

Also, when sarcastic expressions contain positive words, the meaning of sarcasm is often strengthened by emphasizing positive words, so we considered that sentiment scores could be used, but from Table 5.7, it was found that there is a certain correlation between sentiment scores and the probability of not being a detection target, predicted labels, and actual labels, so this hypothesis is considered to have been correct.

6.3 Relationship Between Threshold Values p, s and Detection Accuracy

When all comments analyzed as positive by the Tohoku University BERT model, negative by the fine-tuned model in Section 5.3.1, and with predicted label 0 in binary classification on whether they are detection targets are classified into two groups by actual label, and average values of probability of not being detection target and sentiment score are calculated for each, the result is as shown in Table 6.1. In this study, both threshold values p and s defined in Section 4.1.2 were set to 0.90, but from Table 6.1, for the value of p, making it larger than this increases the possibility of false detection, and conversely making it smaller decreases the Recall value, so it is considered to have been an appropriate value. On the other hand, considering the difference in average sentiment score values, it is considered that for the value of s, the Recall value could have been improved while suppressing the risk of false detection by making it a little smaller.

Table 6.1: Average values of probability of not being detection target and sentiment score

Illustrations are not included in the reading sample

In pattern matching, by using a fine-tuned binary sentiment classification model in addition to syntactic patterns, we were able to achieve improved detection accuracy as shown in Table 5.8, but when the sentiment binary classification model was not used, the results shown in Table 6.2 indicate that the number of false detections was greater than the number that could be correctly detected; therefore, it was found that when using pattern matching, it is important to add conditions such as sentiment analysis results, not just syntactic patterns, to reduce false detections. However, when dealing with sarcastic expressions, existing sentiment analysis models often produce incorrect results, so it is considered that caution is necessary.

Table 6.2: Pattern matching results when sentiment binary classification model was not used

Illustrations are not included in the reading sample

6.5 Future Improvements

The proposed model was ultimately able to perform binary classification on whether comments contain positive words and are intended for sarcasm or abuse with higher accuracy than when the Tohoku University BERT model was simply fine-tuned, but it can be seen that as when comparing sentiment analysis accuracy, there is also a difference in Recall values between these two models. This is because methods using two sentiment analysis results and methods using pattern matching enabled a certain number of comments that are mistakenly considered not detection targets in the existing model to be correctly predicted as detection targets, and since the proposed model maintained the proportion of false detections at about the same level as the existing model while improving detection accuracy for detection target comments, it can be concluded that the proposed model is also superior to the existing model in terms of practicality. However, since the Precision value hardly improved from the existing model, to construct a higher-performance model, it is considered necessary to also set conditions for changing the predicted label to 0 when the predicted label is 1.

Finally, in the method using pattern matching, this study introduced a fine-tuned sentiment binary classification model to reduce false detections, but from the ablation study results, it became clear that using only the method using pattern matching results in a lower Precision value than the existing model; therefore, it is considered that measures to reduce false detections were insufficient. Conversely, from Table 5.10, excluding the Precision value from the method using pattern matching, the proposed model values exceeded the existing model in everything; therefore, the most critical challenge for the future can also be considered to be the reduction of false detections by pattern matching.

Chapter 7

Conclusion

In this study, we attempted to construct a model capable of performing binary classification on whether comments contain positive words and are intended for sarcasm or abuse with higher accuracy than when simply fine-tuning the Tohoku University BERT model by performing several finetunings on the existing Tohoku University BERT model and combining them with pattern matching. As a result, we were able to surpass the existing model in all four evaluation metrics—Accuracy, Precision, Recall, and F1— demonstrating the practicality of the proposed model. However, since we could hardly improve the Precision value, challenges remain.

Since we were able to construct a model with a higher Recall value than the existing model, in the future, considering the ablation study results, we want to particularly focus on the challenge of increased false detections by pattern matching and aim to construct a model where the possibility of false detection is reduced compared to the current situation—that is, not only the Recall value but also the Precision value is improved. To that end, we want to explore new conditions in pattern matching, such as limiting the combinations of keywords within sentences and at sentence ends that correspond to syntactic patterns more than the current situation, not just conditions regarding sentiment analysis results. Furthermore, another future challenge is to construct a model that can achieve similar accuracy even when comment datasets are widely collected using various SNS, not using a skewed dataset.

References

[1] Safer Internet Association. Measures against defamation. March 3, 2023. https://www.soumu.go.jp/main_content/000866255.pdf

[2] Tohoku NLP Group. Pretrained Japanese BERT models. https:// github.com/cl-tohoku/bert-japanese

[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans-formers for language understanding. arXiv:1810.04805,2018.

[4] MeCab: Yet Another Part-of-Speech and Morphological Analyzer. https://taku910.github.io/mecab/

[5] Yonghui Wu,Mike Schuster,Zhifeng Chen,Quoc V. Le,Mohammad Norouzi,Wolfgang Macherey,Maxim Krikun,Yuan Cao,Qin Gao, Klaus Macherey,Jeff Klingner,Apurva Shah,Melvin Johnson,Xiaob- ing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa,Keith Stevens,George Kurian,Nishant Patil,Wei Wang,Cliff Young,Jason Smith,Jason Riesa,Alex Rudnick,Oriol Vinyals,Greg Corrado,Macduff Hughes,and Jeffrey Dean. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144,2016

[6] Hugging Face. https://huggingface.co/

[7] Fumiya Isono,Takashi Matsuyoshi,Fumiyo Fukumoto. Classification and automatic detection of sarcasm on web bulletin boards. IPSJ SIG Technical Report. Natural Language Processing Research Group Report 2013(7),p.1-8

[8] Inui and Suzuki Laboratory. Japanese Sentiment Polarity Dictionary. https://www.cl.ecei.tohoku.ac.jp/Open_Resources-Japanese_ Sentiment_Polarity_Dictionary.html

[9] Satoshi Hiai, Kazutaka Shimada. Analysis and extraction of sarcastic sentences focusing on evaluative expressions. IPSJ Kyushu Branch Hi- nokuni Information Symposium 2016

[10] Shota Suzuki,Ryohei Orihara,Yuichi Sei,Yasuyuki Tahara,and Akihiko Ohsuga. Sarcasm Detection Method to Improve Review Analysis. The 9th International Conference on Agents and Artificial Intelligence (ICAART2017),Volume 2,pp.519-526,2017

[11] Yazhou Zhang,Dan Ma,Prayag Tiwari,Chen Zhang,Mehedi Ma- sud,Mohammad Shorfuzzaman,and Dawei Song. Stance-level Sarcasm Detection with BERT and Stance-centered Graph Attention Networks. ACM Transactions on Internet TechnologyVolume 23,Issue 2. Article No.: 27,pp 1-21,2023

[12] modelpress Entertainment & Lifestyle News Site. YouTube dislike video ranking top 20. November 13, 2021. https://mdpr.jp/news/detail/ 2865682

[...]