Imputation of missing data using masked denoising autoencoder with L2-norm regularization in software effort estimation

Marco, Robert and Syed Ahmad, Sharifah Sakinah (2024) Imputation of missing data using masked denoising autoencoder with L2-norm regularization in software effort estimation. International Journal of Intelligent Engineering and Systems, 17 (4). pp. 299-313. ISSN 2185-3118

Text 00896311220241139231570.pdf Download (980kB)

Abstract

A frequent problem in building initial software effort estimation (SEE) models is the existence of many missing values in historical software engineering datasets. Due to human intervention, this is caused by frequent damage to software project data. Loss of information and bias in data analysis due to missing data are serious problems. This study proposes a method to estimate missing data using a masked-denoising autoencoder (MaskedDAE) with L2-norm regularization, which can handle various types of data, missing patterns, proportions, and distributions. In this study, Cocomo81 and ISBSG-IFPUG datasets from open-source repositories were used. This experiment involved five missing data techniques, eight missing data rates (from 10% to 80%), and two missingness mechanisms (MCAR: missing completely at random and MNAR: missing not at random). The results show that the proposed Mask-DAE method has the best imputation performance in terms of imputation errors by outperforming DAE, k-nearest neighbor imputation (kNNI), random forest (RF) imputation, multiple imputations by chained equation (MICE), mean imputation and mode imputation. We find that the prediction error rate increases with the rate of missing data. Furthermore, prediction errors generated by MCAR mechanisms are lower than those generated by MNAR. Nevertheless, our method can reduce the model variance, which results in lower generalization error.

Item Type:	Article
Uncontrolled Keywords:	Software effort estimation, Missing data imputation, Denoising autoencoder, Missingness mechanisms.
Divisions:	Faculty of Information and Communication Technology
Depositing User:	Norfaradilla Idayu Ab. Ghafar
Date Deposited:	14 Mar 2025 16:16
Last Modified:	14 Mar 2025 16:16
URI:	http://eprints.utem.edu.my/id/eprint/28448
Statistic Details:	View Download Statistic

Actions (login required)

View Item