Hybrid CNN and Autoencoder Deep Learning Model for Network Malware Detection

Abstract

Malware remains one of the primary threats to network security, continuously evolving with increasingly complex attack patterns that are difficult to detect using conventional methods. Data imbalance and high feature dimensionality are major challenges in improving the performance of malware detection models. This study aims to develop a deep learning-based malware detection model using a hybrid approach that combines Convolutional Neural Networks (CNN) and Autoencoders. The dataset used in this study was the improved version of the CICIDS2017 dataset, consisting of more than 2 million records and 91 features. The research stages included data collection, exploratory data analysis (EDA), data preprocessing, feature selection, and data balancing using SMOTE, followed by model design and evaluation. The Autoencoder was employed for dimensionality reduction, generating a compressed representation of 32 features, which was subsequently used as input for the CNN model in multi-class classification. The results demonstrate that the proposed model achieved high accuracy, along with strong precision, recall, and F1-score values across most classes. However, performance on minority classes still exhibited limitations due to data imbalance. Therefore, the hybrid CNN–Autoencoder approach proved effective in improving network malware detection performance.