Binary Opcode Transition Graph Conditioned Malware Family Evolution Tracking Using an Improved PointNeXt Model

Abstract

In this research, we propose an enhanced PointNeXt model for monitoring the growth of malware families under binary opcode transition graph settings. Weighted opcode transition graphs can nonetheless offer some structural information about instruction order, local branching behaviour, and repeated payload routines even in the absence of source code and trustworthy runtime traces. The suggested approach uses transition-aware neighbourhood aggregation with family prototype memory, breaks down binary samples, creates smoothed directed opcode transition graphs, and converts node and transition statistics into graph-point representations. Family categorisation, lineage tracking, open-set rejection, and reviewer-facing threat interpretation are all supported by the system. Packed variants, instruction substitution, padding blocks, mixed obfuscation, and version drift are assessed using a simulated malware corpus of 18,600 binaries from 12 families and 1,900 benign boundary samples. The suggested model obtained 93.8% family classification accuracy, 89.6% evolution tracking recall, and a 17.4% reduction in lineage distance error when compared to the opcode histogram, sequence, graph neural networks, graph Transformers, the original PointNeXt, and PointNeXt without prototype memory. According to ablation research, prototype memory regularises family trajectory tracking, and edge-aware aggregation is more reliable. In this sense, it may be viewed as a structural triage tool for reverse engineers, and multi-architecture binaries, dynamic traces, and actual malware feeds are needed for additional verification.