Abstract: Control flow confusion is a software protection technique. There is uncertainty of obfuscation strength and extra cost by using the traditional garbage code control flow obfuscation algorithm. To solve this problem, a control flow obfuscation algorithm based on nested complexity is proposed. The cost introduced by obfuscation is calculated quantitatively, and the complexity of control flow is measured by nested complexity. The knapsack decision table is constructed based on the idea of packet knapsack. Considering the obfuscation strength and the cost, the garbage code insertion point is selected, and the obfuscation strength is increased as much as possible within the cost threshold. The results show that the obfuscation strength of the algorithm in this paper is higher than that of the control flow obfuscation algorithm using the traditional random insertion strategy.

Key words: control flow obfuscation, nested complexity, packet knapsack, obfuscation strengthh, random insertion