API Call-Based Windows Malware Detection Using CNN-LSTM-Attention and Explainable SHAP

Abstract

The Windows operating system remains a primary target for cyberattacks, with modern malicious software increasingly relying on obfuscation, packing, and polymorphism to evade signature-based detection. This research addresses this challenge by developing a hybrid Convolutional Neural Network, Long Short-Term Memory, and Multi-Head Attention model to classify Windows malicious software from sequences of the first 100 dynamic Application Programming Interface calls. Drawing on the Kaggle "Malware Analysis Datasets: API Call Sequences" refined to 13,206 unique samples with a severe class imbalance of 95.36% malware versus 4.64% benign, the model employs a class-weighting strategy to ensure robust learning. Rigorous evaluation through five-fold cross-validation demonstrates that the proposed architecture achieves a mean Area Under the Receiver Operating Characteristic Curve of 0.9562 (± 0.0186) and an accuracy of 95.78% (± 1.05%). Ablation studies confirm that the Multi-Head Attention mechanism is statistically indispensable (p < 0.05), as its removal degrades performance to an Area Under the Receiver Operating Characteristic Curve of 0.9073. While competitive with ensemble baselines such as Random Forest and XGBoost, the proposed approach provides superior interpretability by integrating SHapley Additive Explanations. This enables local and global forensic accountability by pinpointing critical temporal execution windows specifically at steps t₁, t₅₀, and t­₇₅ that drive malicious predictions. These results indicate that combining hybrid deep learning with post-hoc explainability produces malware detection that is both robust and forensically transparent.

GS Cited Analysis