Original Article
Split Viewer
International Journal of Fuzzy Logic and Intelligent Systems 2024; 24(4): 317-332
Published online December 25, 2024
https://doi.org/10.5391/IJFIS.2023.24.4.317
© The Korean Institute of Intelligent Systems
Optimizing Deep Learning Models with Hybrid Nonlinear Loss Functions: Integrating Heterogeneous Nonlinear Properties for Enhanced Performance
Seoung-Ho Choi
Faculty of Basic Liberal Art, College of Liberal Arts, Hansung University, Seoul, Korea
Correspondence to :
Seoung-Ho Choi (jcn99250@naver.com)
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current deep learning models underperform when using loss functions characterized by single properties. Therefore, optimizing these models with a combination of multiple attributes is essential to enhance performance. We propose a novel hybrid nonlinear loss function technique incorporating heterogeneous nonlinear properties to achieve optimal performance. We evaluated the proposed method using six analytical techniques: contour map visualization, loss error frequency analysis, scatter plot visualization, loss function visualization, gradient descent analysis, and covariate Analysis with convex and linear coefficients. Our experiments on semantic segmentation tasks using the Pascal visual object classes and automatic target recognition datasets demonstrated superior optimization performance compared to existing loss functions.
Keywords: Hybrid nonlinear loss function, Heterogeneous properties, Optimization, High performance
Acknowledgments
Conflict of Interest
No potential conflict of interest relevant to this article was reported.
PDF
Standard view
Export citation
Share Article
Original Article
International Journal of Fuzzy Logic and Intelligent Systems 2024; 24(4): 317-332
Published online December 25, 2024 https://doi.org/10.5391/IJFIS.2023.24.4.317
Copyright © The Korean Institute of Intelligent Systems.
Optimizing Deep Learning Models with Hybrid Nonlinear Loss Functions: Integrating Heterogeneous Nonlinear Properties for Enhanced Performance
Seoung-Ho Choi
Faculty of Basic Liberal Art, College of Liberal Arts, Hansung University, Seoul, Korea
Correspondence to:Seoung-Ho Choi (jcn99250@naver.com)
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract
Current deep learning models underperform when using loss functions characterized by single properties. Therefore, optimizing these models with a combination of multiple attributes is essential to enhance performance. We propose a novel hybrid nonlinear loss function technique incorporating heterogeneous nonlinear properties to achieve optimal performance. We evaluated the proposed method using six analytical techniques: contour map visualization, loss error frequency analysis, scatter plot visualization, loss function visualization, gradient descent analysis, and covariate Analysis with convex and linear coefficients. Our experiments on semantic segmentation tasks using the Pascal visual object classes and automatic target recognition datasets demonstrated superior optimization performance compared to existing loss functions.
Keywords: Hybrid nonlinear loss function, Heterogeneous properties, Optimization, High performance
Fig 1.
Download
Proposed method: (a) hybrid loss function case 1 and (b) hybrid loss function case 2.
Fig 2.
Download
Visualization of used loss function: (a) no loss function, (b) L1 and L2 loss functions, (c) nonlinear exponential loss function, (d) nonlinear exponential loss function and L1/L2 loss functions, (e) exponential moving average loss function, and (f) exponential moving average loss function and nonlinear loss function.
Fig 3.
Download
Visualization of experiment method configuration.
Fig 4.
Download
Visualization of the analysis of the proposed method using contour maps of 6 different loss functions with balanced and unbalanced input data: (a) existing experimental equation, (b) L1 and L2 loss function addition formulas, (c) nonlinear exponential loss function, (d) nonlinear exponential loss function and linear combination L1 & L2 loss functions, (e) exponential moving average loss function, and (f) exponential moving average loss function and nonlinear exponential loss function.
Fig 5.
Download
Visualization of the impact of 6 different loss functions on the frequency distribution of generated values with unbalanced and balanced input data: (a) existing experimental equation, (b) L1 and L2 loss function addition formulas, (c) nonlinear exponential loss function, (d) nonlinear exponential loss function and linear combination L1 & L2 loss functions, (e) exponential moving average loss function, and (f) exponential moving average loss function and nonlinear exponential loss function
Fig 6.
Download
Visualization of scatter plots using LogisticGroupLasso model to analyze the model and sparsity values of 6 different loss functions: (a) existing experimental equation, (b) L1 and L2 loss function addition formulas, (c) nonlinear exponential loss function, (d) nonlinear exponential loss function and linear combination L1 & L2 loss functions, (e) exponential moving average loss function, and (f) exponential moving average loss function and nonlinear exponential loss function.
Fig 7.
Download
Visualization of 3D surface for 9 equations: (a)
Fig 8.
Download
Visualization of heatmap for 9 equations: (a)
Fig 9.
Download
Visualization of comparison of the loss function methods: (a) loss values for different training epochs and (b) IOU values for different training epochs.
-
Table 1 . The index definition of hybrid loss function.
Name Index A None Experiment 1 Combination L1 and L2 loss function with convex coefficient Experiment 2 Combination L1 and L2 loss function with linear coefficient Experiment 3 Nonlinear exponential loss function with convex coefficient Experiment 4 Nonlinear exponential loss function with linear coefficient Experiment 5 Exponential hybrid loss function with convex coefficient Experiment 6 Exponential hybrid loss function with linear coefficient Experiment 7 Exponential moving average loss function with convex coefficient adopted no filter Experiment 8 Exponential moving average loss function with convex coefficient adopted ReLU Experiment 9 Exponential moving average loss function with convex coefficient adopted eLU Experiment 10 Exponential moving average loss function with convex coefficient adopted Bi-ReLU Experiment 11 Exponential moving average loss function with convex coefficient adopted Bi-eLU Experiment 12 Exponential moving average loss function with linear coefficient adopted no filter Experiment 13 Exponential moving average loss function with linear coefficient adopted ReLU Experiment 14 Exponential moving average loss function with linear coefficient adopted eLU Experiment 15 Exponential moving average loss function with linear coefficient adopted Bi-ReLU Experiment 16 Exponential moving average loss function with linear coefficient adopted Bi-eLU Experiment 17 Exponential hybrid v2 loss function with convex coefficient adopted no filter Experiment 18 Exponential hybrid v2 loss function with convex coefficient adopted ReLU Experiment 19 Exponential hybrid v2 loss function with convex coefficient adopted eLU Experiment 20 Exponential hybrid v2 loss function with convex coefficient adopted Bi-ReLU Experiment 21 Exponential hybrid v2 loss function with convex coefficient adopted Bi-eLU Experiment 22 Exponential hybrid v2 loss function with linear coefficient adopted no filter Experiment 23 Exponential hybrid v2 loss function with linear coefficient adopted ReLU Experiment 24 Exponential hybrid v2 loss function with linear coefficient adopted eLU Experiment 25 Exponential hybrid v2 loss function with linear coefficient adopted Bi-ReLU Experiment 26 Exponential hybrid v2 loss function with linear coefficient adopted Bi-eLU Experiment 27
-
Table 2 . Experiment result of loss function using U-Net on ATR dataset with Seed 250.
Loss function DSC F1-score IOU Loss Precision Recall Experiment1 0.724 0.724 0.768 1.072 0.724 0.724 Experiment2 0.723 0.723 0.767 1.525 0.723 0.723 Experiment3 0.724 0.724 0.768 1.398 0.724 0.724 Experiment4 0.724 0.724 0.768 1.364 0.724 0.724 Experiment5 0.724 0.724 0.767 1.287 0.724 0.724 Experiment6 0.723 0.723 0.767 1.222 0.723 0.723 Experiment7 0.724 0.724 0.768 1.278 0.724 0.724 Experiment8 0.724 0.724 0.768 1.255 0.724 0.724 Experiment9 0.724 0.724 0.768 1.228 0.724 0.724 Experiment10 0.724 0.724 0.768 1.237 0.724 0.724 Experiment11 0.724 0.724 0.768 1.195 0.724 0.724 Experiment12 0.724 0.724 0.768 1.164 0.724 0.724 Experiment13 0.724 0.724 0.768 1.136 0.724 0.724 Experiment14 0.724 0.724 0.768 1.112 0.724 0.724 Experiment15 0.724 0.724 0.768 1.115 0.724 0.724 Experiment16 0.724 0.724 0.768 1.095 0.724 0.724 Experiment17 0.724 0.724 0.768 1.08 0.724 0.724 Experiment18 0.725 0.724 0.768 1.075 0.724 0.724 Experiment19 0.725 0.725 0.768 1.09 0.725 0.725 Experiment20 0.725 0.725 0.768 1.077 0.725 0.725 Experiment21 0.725 0.725 0.768 1.069 0.725 0.725 Experiment22 0.725 0.725 0.768 1.07 0.725 0.725 Experiment23 0.725 0.725 0.768 1.077 0.725 0.725 Experiment24 0.725 0.725 0.768 1.07 0.725 0.725 Experiment25 0.725 0.725 0.768 1.052 0.725 0.725 Experiment26 0.725 0.725 0.768 1.044 0.725 0.725 Experiment27 0.725 0.725 0.768 1.042 0.725 0.725
-
Algorithm 1. Hybrid loss function case 1.
while Epoch ≠ 0do Loss ←Crossentropy loss function + Nonlinear exponential lossfunction + L 1lossfunction +L 2lossfunction Optimizationmodel using Loss error end while
-
Algorithm 2. Hybrid loss function case 2.
while Epoch ≠ 0do Loss ←Crossentropy loss function + Exponential moving average + Nonlinear exponential lossfunction Optimizationmodel using Loss error end while
Article Tools
Metrics
Share this article on :
Related articles in IJFIS
-
An Artificial Neural Network-Based RFID Network Planning Method for Asset Monitoring in Healthcare
Le Van Hoa and Vo Viet Minh Nhat
International Journal of Fuzzy Logic and Intelligent Systems 2024; 24(3): 181-193
https://doi.org/10.5391/IJFIS.2024.24.3.181
-
Simultaneous Kernel Learning and Label Imputation for Pattern Classification with Partially Labeled Data
Minyoung Kim
Int. J. Fuzzy Log. Intell. Syst. 2017; 17(1): 10-16
https://doi.org/10.5391/IJFIS.2017.17.1.10
-
Document Summarization via Convex-Concave Programming
Minyoung Kim
Int. J. Fuzzy Log. Intell. Syst. 2016; 16(4): 293-298
https://doi.org/10.5391/IJFIS.2016.16.4.293
An Artificial Neural Network-Based RFID Network Planning Method for Asset Monitoring in Healthcare
Le Van Hoa and Vo Viet Minh Nhat
International Journal of Fuzzy Logic and Intelligent Systems 2024; 24(3): 181-193 https://doi.org/10.5391/IJFIS.2024.24.3.181Simultaneous Kernel Learning and Label Imputation for Pattern Classification with Partially Labeled Data
Minyoung Kim
Int. J. Fuzzy Log. Intell. Syst. 2017; 17(1): 10-16 https://doi.org/10.5391/IJFIS.2017.17.1.10Document Summarization via Convex-Concave Programming
Minyoung Kim
Int. J. Fuzzy Log. Intell. Syst. 2016; 16(4): 293-298 https://doi.org/10.5391/IJFIS.2016.16.4.293
