Designing an optimizing strategy for malware analysis using innovative techniques from data mining

Abstract

Malware an all-time threat to the computing world, goes through many revolutions to evade antimalware mechanisms. Protecting a host from untrusted programs, as well as known/unknown malware, has evolved as a challenge in front of researchers. Hence the search for optimal approaches are being continued, which may be computationally feasible and have higher accuracy by lowering the false positive and false negative cases. This research investigates both misuse and anomaly detection approaches. This dissertation uses a novel approach named LCSBV (Longest Common Subsequence Based majority Voting) and a Graph structure based optimized approach by extending concepts from data mining to detect new malware and host specific anomalous processes respectively. newlineThe proposed methodology towards misuse detection explores a static feature named PSI (Printable String Information) sequence, dynamic feature named fixed length API (Application Programming Interface) sequence and hybrid feature set for analysis and detection of existing as well as new malware. The models are trained with Random Forest (RF) and Support Vector Machine (SVM) for each of the three feature sets individually. It is found that SVM performed better than RF with accuracy 96.2% for static feature set of PSI, 97.4% for dynamic feature set of fixed length API sequence and 98.2% for the combined feature set. A dynamic feature set of variable length API sequence is used to generate a Signature Semantic Base for detecting malware. newline