Machine Learning in Cyber Defence

Why machine learning sits at the heart of cyber defence

Rules and signatures struggle with threats designed to change. Machine learning (ML) works differently: it learns patterns and behaviours from data, so it can pick up subtle signs of attack that analysts or static rules would miss. The book highlights three contributions.

Adoption has been gradual. Basic anomaly detection in intrusion detection systems emerged in the early 2000s; big data in the 2010s enabled richer supervised and unsupervised models; deep learning, natural language processing and cloud computing then pushed ML into phishing prevention, malware detection and behavioural analytics. The obstacles are practical: good labelled data is scarce, over-sensitive models flood teams with false positives, and real-time analysis demands serious computing power.

• Threat prediction: behavioural baselines for users, devices and networks, with instant alerts on deviations.
• Continuous learning: models that evolve with new data, improving detection of unknown and zero-day threats.
• Scale: real-time monitoring of data volumes no human team or conventional tool could process.

Supervised learning: recognising the known

Supervised models learn from labelled examples of malicious and benign activity, then classify new data. Decision trees offer interpretable logic, random forests cope well with noisy data, support vector machines weigh many indicators at once, and neural networks capture complex patterns, even spotting malware by converting binary files into images. Typical uses include malware classification, phishing detection, intrusion detection and spam filtering.

Their weakness is dependence on the past. Labelling data is slow, specialist work; poorly tuned models trigger false alarms or overfit to familiar data; and genuinely novel attacks may go unrecognised. The book points to transfer learning, semi-supervised learning and active learning, where the model asks experts to label ambiguous cases, as ways to stretch limited labelled data further.

Unsupervised learning: finding the unknown

Unsupervised learning needs no labels. Clustering groups similar activity and highlights what does not fit, which makes it a cornerstone of proactive defence against zero-day exploits, APTs and insider attacks. K-means is fast but needs the number of clusters set in advance; DBSCAN finds outliers in sparse data; hierarchical clustering explores data with unknown structure; and isolation forests efficiently isolate rare, high-risk events.

Applications span network traffic analysis, user behaviour analytics, intrusion detection, malware grouping and endpoint monitoring. A prolonged, high-volume transfer at an unusual hour, for example, may signal data exfiltration or command-and-control activity. The trade-offs are false positives (a late-working employee can look suspicious), heavy computational demands and limited interpretability, which is why the book stresses adding context and explainable AI.

Reinforcement learning and hybrid defence

Reinforcement learning (RL) learns by trial and error. An agent, such as an endpoint detection and response (EDR) tool, observes the state of its environment, takes actions such as isolating a device or blocking traffic, and receives rewards or penalties that refine its policy over time. This suits intrusion response, automated threat hunting, adaptive malware defence and tuning security policies to balance protection against disruption. The risks are distinctive: poorly designed rewards can make responses too aggressive or too timid, and excessive exploration can itself expose vulnerabilities.

Because no single technique is enough, the book argues for hybrid approaches.

• Supervised plus unsupervised learning to catch both familiar and unfamiliar threats.
• Reinforcement plus supervised learning to recognise known attacks and adapt the response.
• Rules plus machine learning: rules block the known, ML catches the evolving.
• Multi-model ensembles of decision trees, random forests and neural networks for accuracy and robustness.

Predictive models and malware classification

Predictive analysis estimates the likelihood of future incidents from historical and real-time data. Decision trees are fast and transparent but prone to overfitting; neural networks handle complex, high-dimensional data but behave as “black boxes”, a real problem in compliance-sensitive environments. Ensembles often use trees for rapid filtering and neural networks for deeper analysis.

In malware classification, feature quality matters most. Static features include file headers, embedded strings and API calls; behavioural features cover rapid file encryption, registry changes and process injection; network features capture connections to unusual addresses, abnormal transfer rates and suspicious DNS or HTTP requests. The book's case studies include convolutional neural networks catching polymorphic malware from its visual pattern, and isolation forests helping a financial services firm contain ransomware within minutes.

User behaviour analytics and insider threats

Insiders already hold legitimate access, so the book advocates user behaviour analytics (UBA): establishing a baseline for each user, then watching for change. Clustering, isolation forests, time-series analysis, Markov chains, recurrent neural networks, local outlier factor and autoencoders all play a part. Key indicators include the following.

In one financial services example, the system flagged an analyst downloading large volumes of client transaction data late at night, stopping a planned transfer to an unauthorised external party. Monitoring employees raises genuine privacy questions, so organisations must balance security with data protection law; federated learning and explainable AI are the book's routes to doing so responsibly.

• Unusual login times, locations or frequency.
• Access to sensitive data beyond a user's role.
• Large downloads, deletions or erased logs.
• Unusual emailing of sensitive or encrypted files.
• Non-technical staff using administrative tools.
• Out-of-hours physical access revealed by badge data.