International Journal of Fuzzy Logic and Intelligent Systems 2022; 22(4): 401-413
Published online December 25, 2022
https://doi.org/10.5391/IJFIS.2022.22.4.401
© The Korean Institute of Intelligent Systems
Alif Tri Handoyo1*, Hidayaturrahman1, Criscentia Jessica Setiadi2, Derwin Suhartono1
1Computer Science Department, School of Computer Science, Bina Nusantara University, Jakarta, Indonesia, 11480
2English Department, Faculty of Humanities, Bina Nusantara University, Jakarta, Indonesia, 11480
Correspondence to :
Alif Tri Handoyo (alif.handoyo@binus.ac.id)
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Sarcasm is the use of words commonly used to ridicule someone or for humorous purposes. Several studies on sarcasm detection have utilized different learning algorithms. However, most of these learning models have always focused on the contents of expression only, thus leaving the contextual information in isolation. As a result, they failed to capture the contextual information in the sarcastic expression. Moreover, some datasets used in several studies have an unbalanced dataset, thus impacting the model result. In this paper, we propose a contextual model for sarcasm identification in Twitter using various pre-trained models and augmenting the dataset by applying Global Vector representation (GloVe) for the construction of word embedding and context learning to generate more sarcastic data, and also perform additional experiments by using the data duplication method. Data augmentation and duplication impact is tested in various datasets and augmentation sizes. In particular, we achieved the best performance after using the data augmentation method to increase 20% of the data labeled as sarcastic and improve the performance by 2.1% with an F1 Score of 40.44% compared to 38.34% before using data augmentation in the iSarcasm dataset.
Keywords: Twitter sarcasm detection, RoBERTa, Word embedding, Data augmentation
No potential conflict of interest relevant to this article was reported.
International Journal of Fuzzy Logic and Intelligent Systems 2022; 22(4): 401-413
Published online December 25, 2022 https://doi.org/10.5391/IJFIS.2022.22.4.401
Copyright © The Korean Institute of Intelligent Systems.
Alif Tri Handoyo1*, Hidayaturrahman1, Criscentia Jessica Setiadi2, Derwin Suhartono1
1Computer Science Department, School of Computer Science, Bina Nusantara University, Jakarta, Indonesia, 11480
2English Department, Faculty of Humanities, Bina Nusantara University, Jakarta, Indonesia, 11480
Correspondence to:Alif Tri Handoyo (alif.handoyo@binus.ac.id)
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Sarcasm is the use of words commonly used to ridicule someone or for humorous purposes. Several studies on sarcasm detection have utilized different learning algorithms. However, most of these learning models have always focused on the contents of expression only, thus leaving the contextual information in isolation. As a result, they failed to capture the contextual information in the sarcastic expression. Moreover, some datasets used in several studies have an unbalanced dataset, thus impacting the model result. In this paper, we propose a contextual model for sarcasm identification in Twitter using various pre-trained models and augmenting the dataset by applying Global Vector representation (GloVe) for the construction of word embedding and context learning to generate more sarcastic data, and also perform additional experiments by using the data duplication method. Data augmentation and duplication impact is tested in various datasets and augmentation sizes. In particular, we achieved the best performance after using the data augmentation method to increase 20% of the data labeled as sarcastic and improve the performance by 2.1% with an F1 Score of 40.44% compared to 38.34% before using data augmentation in the iSarcasm dataset.
Keywords: Twitter sarcasm detection, RoBERTa, Word embedding, Data augmentation
Workflow of the proposed methodology.
Data augmentation process.
True positive quantity change in percentage after performing data augmentation across multiple datasets.
True negative quantity change in percentage after performing data augmentation across multiple datasets.
False positive quantity change in percentage after performing data augmentation across multiple datasets.
False negative quantity change in percentage after performing data augmentation across multiple datasets.
Table 1 . Distribution of Training and Test data in various datasets.
Dataset | Train | Test | Total |
---|---|---|---|
iSarcasm | 3,463 | 887 | 4,350 |
Ghosh | 37,082 | 4,121 | 41,203 |
Ptacek | 56,677 | 6,298 | 62,975 |
SemEval-18 | 3,776 | 780 | 4,556 |
Table 2 . The proportion of sarcastic and non-sarcastic data in various datasets.
Dataset | Non-sarcastic | Sarcastic | % Sarcasm |
---|---|---|---|
iSarcasm | 3,584 | 766 | 17.62% |
Ghosh | 22,725 | 18,478 | 44.84% |
Ptacek | 31,810 | 31,165 | 49.50% |
SemEval-18 | 2,379 | 2,177 | 49.12% |
Table 3 . The amount of sarcastic data after performing data augmentation in each dataset.
Dataset | Sarcastic data with (10% augmentation) | Sarcastic data with (20% augmentation) | Sarcastic data with (30% augmentation) |
---|---|---|---|
iSarcasm | 820 | 875 | 929 |
Ghosh | 20,147 | 21,816 | 23,485 |
Ptacek | 33,970 | 36,776 | 39,581 |
SemEval-18 | 2,363 | 2,550 | 2,736 |
Table 4 . Augmented text results.
Original text | Augmented text |
---|---|
good morning, please go and vote ! it only takes minutes and a low turnout will hand victory to the brexit party e uelections 2019 | good morning, please go and vote! it only takes left and very low turnout will right victory this the brexit party u uelections 2019 |
Table 5 . Hyper-parameter settings.
Hyper-Parameter | Value |
---|---|
max_seq_length | 40 |
learning_rate | 0.00001 |
weight_decay | 0.01 |
warmup_ratio | 0.2 |
max_grad_norm | 1.0 |
num_train_epochs | 8,13 |
train_batch_size | 16,32 |
fp16 | true |
manual_seed | 128 |
Table 6 . Performance Comparison in terms of F1 Score and MCC for three different models on four different datasets, before and after augmentation with various sizes and with data duplication technique.
Model | Dataset | Before augmentation | Augmentation 10% | Augmentation 20% | Augmentation 30% | Data duplication 20% | |||||
---|---|---|---|---|---|---|---|---|---|---|---|
F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | ||
BERT | iSarcasm | 0.2421 | 0.3450 | 0.2499 | 0.3217 | 0.2567 | 0.2922 | 0.2410 | 0.3406 | ||
Ghosh | 0.7846 | 0.6184 | 0.7760 | 0.6042 | 0.7693 | 0.5963 | 0.7732 | 0.6048 | |||
Ptacek | 0.8596 | 0.7184 | 0.8602 | 0.7192 | 0.8598 | 0.7190 | 0.8587 | 0.7165 | |||
SemEval-18 | 0.6442 | 0.3642 | 0.6503 | 0.3825 | 0.6257 | 0.3505 | 0.6259 | 0.3578 | |||
RoBERTa | iSarcasm | 0.3834 | 0.2842 | 0.3809 | 0.2964 | 0.3828 | 0.2939 | 0.3925 | 0.2914 | ||
Ghosh | 0.7904 | 0.6299 | 0.7830 | 0.6284 | 0.7758 | 0.6193 | 0.7835 | 0.6294 | |||
Ptacek | 0.8735 | 0.7454 | 0.8738 | 0.7491 | 0.8727 | 0.7469 | 0.8717 | 0.7442 | |||
SemEval-18 | 0.6637 | 0.4109 | 0.6666 | 0.4286 | 0.6707 | 0.4362 | 0.6627 | 0.4134 | |||
DistilBERT | iSarcasm | 0.3083 | 0.2080 | 0.2924 | 0.1890 | 0.2784 | 0.1926 | 0.2991 | 0.2224 | ||
Ghosh | 0.7831 | 0.6148 | 0.7651 | 0.5854 | 0.7620 | 0.5799 | 0.7571 | 0.5700 | |||
Ptacek | 0.8542 | 0.7101 | 0.8538 | 0.7094 | 0.8508 | 0.7055 | 0.8569 | 0.7163 | |||
SemEval-18 | 0.6066 | 0.3180 | 0.6240 | 0.3534 | 0.6366 | 0.3822 | 0.6130 | 0.3261 |
Table 7 . Comparison table of similar work.
Authors | Techniques used | Discussions |
---|---|---|
Xu Guo et al. [4] | Model: BERT with Latent-Optimization Method | F1 Score: |
Amirhossein et al. [16] | Model: BERT-based Data | F1 Score: |
Our proposed method | Model: BERT, RoBERTa, DistilBERT | F1 Score: |
Ho-Seung Kim and Jee-Hyong Lee
International Journal of Fuzzy Logic and Intelligent Systems 2024; 24(2): 83-92 https://doi.org/10.5391/IJFIS.2024.24.2.83Amit Kumar Sharma, Rakshith Alaham Gangeya, Harshit Kumar, Sandeep Chaurasia, and Devesh Kumar Srivastava
International Journal of Fuzzy Logic and Intelligent Systems 2022; 22(3): 325-338 https://doi.org/10.5391/IJFIS.2022.22.3.325Yagya Raj Pandeya, and Joonwhoan Lee
Int. J. Fuzzy Log. Intell. Syst. 2018; 18(2): 154-160 https://doi.org/10.5391/IJFIS.2018.18.2.154Workflow of the proposed methodology.
|@|~(^,^)~|@|Data augmentation process.
|@|~(^,^)~|@|True positive quantity change in percentage after performing data augmentation across multiple datasets.
|@|~(^,^)~|@|True negative quantity change in percentage after performing data augmentation across multiple datasets.
|@|~(^,^)~|@|False positive quantity change in percentage after performing data augmentation across multiple datasets.
|@|~(^,^)~|@|False negative quantity change in percentage after performing data augmentation across multiple datasets.