Article Search
닫기

Original Article

Split Viewer

International Journal of Fuzzy Logic and Intelligent Systems 2022; 22(4): 401-413

Published online December 25, 2022

https://doi.org/10.5391/IJFIS.2022.22.4.401

© The Korean Institute of Intelligent Systems

Sarcasm Detection in Twitter- Performance Impact While Using Data Augmentation: Word Embeddings

Alif Tri Handoyo1*, Hidayaturrahman1, Criscentia Jessica Setiadi2, Derwin Suhartono1

1Computer Science Department, School of Computer Science, Bina Nusantara University, Jakarta, Indonesia, 11480
2English Department, Faculty of Humanities, Bina Nusantara University, Jakarta, Indonesia, 11480

Correspondence to :
Alif Tri Handoyo (alif.handoyo@binus.ac.id)

Received: January 27, 2022; Revised: July 16, 2022; Accepted: December 19, 2022

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Sarcasm is the use of words commonly used to ridicule someone or for humorous purposes. Several studies on sarcasm detection have utilized different learning algorithms. However, most of these learning models have always focused on the contents of expression only, thus leaving the contextual information in isolation. As a result, they failed to capture the contextual information in the sarcastic expression. Moreover, some datasets used in several studies have an unbalanced dataset, thus impacting the model result. In this paper, we propose a contextual model for sarcasm identification in Twitter using various pre-trained models and augmenting the dataset by applying Global Vector representation (GloVe) for the construction of word embedding and context learning to generate more sarcastic data, and also perform additional experiments by using the data duplication method. Data augmentation and duplication impact is tested in various datasets and augmentation sizes. In particular, we achieved the best performance after using the data augmentation method to increase 20% of the data labeled as sarcastic and improve the performance by 2.1% with an F1 Score of 40.44% compared to 38.34% before using data augmentation in the iSarcasm dataset.

Keywords: Twitter sarcasm detection, RoBERTa, Word embedding, Data augmentation

This work is supported by Research and Technology Transfer Office, Bina Nusantara University as a part of Bina Nusantara University’s International Research Grant entitled Sarcasm Detection in Twitter - Performance Impact while using Data Augmentation: Word Embeddings with contract number: 061/VR.RTT/IV/2022 and contract date: 8 April 2022.

No potential conflict of interest relevant to this article was reported.

Alif Tri Handoyo received his S.T. (Informatics Engineering) from Hasanuddin University, Indonesia (2021). Currently ƒhe is a graduate student at Bina Nusantara University (BINUS). His research interest includes natural language processing, image processing and internet of things.

Hidayaturrahman is faculty member of Bina Nusantara University. He got his master’s degree from Bandung Institute of Technology in 2018. His research fields are Computer Vision, Natural Language Processing, and Machine Learning.

Criscentia Jessica Setiadi is a faculty member of English Literature department, Bina Nusantara University. Her research interests include adaptation, transmedia, and comparative literature studies. She is currently pursuing her doctoral degree on cultural studies in University of Indonesia, reading social media and mediatization.

Derwin Suhartono is faculty member of Bina Nusantara University, Indonesia. He got his Ph.D. degree in computer science from Universitas Indonesia in 2018. His research fields are natural language processing. Recently, he is continually doing research in argumentation mining and personality recognition.

Article

Original Article

International Journal of Fuzzy Logic and Intelligent Systems 2022; 22(4): 401-413

Published online December 25, 2022 https://doi.org/10.5391/IJFIS.2022.22.4.401

Copyright © The Korean Institute of Intelligent Systems.

Sarcasm Detection in Twitter- Performance Impact While Using Data Augmentation: Word Embeddings

Alif Tri Handoyo1*, Hidayaturrahman1, Criscentia Jessica Setiadi2, Derwin Suhartono1

1Computer Science Department, School of Computer Science, Bina Nusantara University, Jakarta, Indonesia, 11480
2English Department, Faculty of Humanities, Bina Nusantara University, Jakarta, Indonesia, 11480

Correspondence to:Alif Tri Handoyo (alif.handoyo@binus.ac.id)

Received: January 27, 2022; Revised: July 16, 2022; Accepted: December 19, 2022

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Sarcasm is the use of words commonly used to ridicule someone or for humorous purposes. Several studies on sarcasm detection have utilized different learning algorithms. However, most of these learning models have always focused on the contents of expression only, thus leaving the contextual information in isolation. As a result, they failed to capture the contextual information in the sarcastic expression. Moreover, some datasets used in several studies have an unbalanced dataset, thus impacting the model result. In this paper, we propose a contextual model for sarcasm identification in Twitter using various pre-trained models and augmenting the dataset by applying Global Vector representation (GloVe) for the construction of word embedding and context learning to generate more sarcastic data, and also perform additional experiments by using the data duplication method. Data augmentation and duplication impact is tested in various datasets and augmentation sizes. In particular, we achieved the best performance after using the data augmentation method to increase 20% of the data labeled as sarcastic and improve the performance by 2.1% with an F1 Score of 40.44% compared to 38.34% before using data augmentation in the iSarcasm dataset.

Keywords: Twitter sarcasm detection, RoBERTa, Word embedding, Data augmentation

Fig 1.

Figure 1.

Workflow of the proposed methodology.

The International Journal of Fuzzy Logic and Intelligent Systems 2022; 22: 401-413https://doi.org/10.5391/IJFIS.2022.22.4.401

Fig 2.

Figure 2.

Data augmentation process.

The International Journal of Fuzzy Logic and Intelligent Systems 2022; 22: 401-413https://doi.org/10.5391/IJFIS.2022.22.4.401

Fig 3.

Figure 3.

True positive quantity change in percentage after performing data augmentation across multiple datasets.

The International Journal of Fuzzy Logic and Intelligent Systems 2022; 22: 401-413https://doi.org/10.5391/IJFIS.2022.22.4.401

Fig 4.

Figure 4.

True negative quantity change in percentage after performing data augmentation across multiple datasets.

The International Journal of Fuzzy Logic and Intelligent Systems 2022; 22: 401-413https://doi.org/10.5391/IJFIS.2022.22.4.401

Fig 5.

Figure 5.

False positive quantity change in percentage after performing data augmentation across multiple datasets.

The International Journal of Fuzzy Logic and Intelligent Systems 2022; 22: 401-413https://doi.org/10.5391/IJFIS.2022.22.4.401

Fig 6.

Figure 6.

False negative quantity change in percentage after performing data augmentation across multiple datasets.

The International Journal of Fuzzy Logic and Intelligent Systems 2022; 22: 401-413https://doi.org/10.5391/IJFIS.2022.22.4.401

Table 1 . Distribution of Training and Test data in various datasets.

DatasetTrainTestTotal
iSarcasm3,4638874,350
Ghosh37,0824,12141,203
Ptacek56,6776,29862,975
SemEval-183,7767804,556

Table 2 . The proportion of sarcastic and non-sarcastic data in various datasets.

DatasetNon-sarcasticSarcastic% Sarcasm
iSarcasm3,58476617.62%
Ghosh22,72518,47844.84%
Ptacek31,81031,16549.50%
SemEval-182,3792,17749.12%

Table 3 . The amount of sarcastic data after performing data augmentation in each dataset.

DatasetSarcastic data with (10% augmentation)Sarcastic data with (20% augmentation)Sarcastic data with (30% augmentation)
iSarcasm820875929
Ghosh20,14721,81623,485
Ptacek33,97036,77639,581
SemEval-182,3632,5502,736

Table 4 . Augmented text results.

Original textAugmented text
good morning, please go and vote ! it only takes minutes and a low turnout will hand victory to the brexit party e uelections 2019good morning, please go and vote! it only takes left and very low turnout will right victory this the brexit party u uelections 2019

Table 5 . Hyper-parameter settings.

Hyper-ParameterValue
max_seq_length40
learning_rate0.00001
weight_decay0.01
warmup_ratio0.2
max_grad_norm1.0
num_train_epochs8,13
train_batch_size16,32
fp16true
manual_seed128

Table 6 . Performance Comparison in terms of F1 Score and MCC for three different models on four different datasets, before and after augmentation with various sizes and with data duplication technique.

ModelDatasetBefore augmentationAugmentation 10%Augmentation 20%Augmentation 30%Data duplication 20%

F1MCCF1MCCF1MCCF1MCCF1MCC
BERTiSarcasm0.36040.24210.34500.24990.32170.25670.29220.24100.34060.2817
Ghosh0.78460.61840.77600.60420.76930.59630.77320.60480.80420.6541
Ptacek0.85960.71840.86020.71920.85980.71900.85870.71650.86430.7263
SemEval-180.64420.36420.65030.38250.67060.43080.62570.35050.62590.3578

RoBERTaiSarcasm0.38340.28420.38090.29640.40440.30840.38280.29390.39250.2914
Ghosh0.79040.62990.78300.62840.77580.61930.78350.62940.81080.6691
Ptacek0.87350.74540.87380.74910.87270.74690.87170.74420.87410.7474
SemEval-180.66370.41090.66660.42860.67070.43620.67460.43820.66270.4134

DistilBERTiSarcasm0.30830.20800.29240.18900.27840.19260.33990.24600.29910.2224
Ghosh0.78310.61480.76510.58540.76200.57990.75710.57000.79720.6403
Ptacek0.85420.71010.85380.70940.85080.70550.85690.71630.85570.7118
SemEval-180.60660.31800.62400.35340.65220.40730.63660.38220.61300.3261

Table 7 . Comparison table of similar work.

AuthorsTechniques usedDiscussions
Xu Guo et al. [4]Model: BERT with Latent-Optimization MethodDataset: iSarcasm, PtacekF1 Score:iSarcasm: 34.92%SemEval-18: 66.26%Limitation: no data augmentation was performed to balance the iSarcasm dataset.
Amirhossein et al. [16]Model: BERT-based DataAugmentation: word removalDataset: iSarcasmF1 Score:iSarcasm: 41.4%Limitation: data augmentation only use word removal, thus reducing sentence quality and potentially remove sarcastic nature of sentence
Our proposed methodModel: BERT, RoBERTa, DistilBERTF1 Score:iSarcasm: 40.44%Ghosh: 81.08%Ptacek: 87.41%SemEval-18: 67.46%Novelty: data augmentation using GloVe word embedding, data duplication technique and deeper analysis of data augmentation results

Share this article on :

Related articles in IJFIS