SAT is Dead

By: Alex Henthorn-Iwane

Security awareness training (SAT) has been around for over a decade and is now common practice. Today, most responsible corporations run an SAT program. That might seem like a victory for internet security, and in a sense, it is. Yet, from the point of view of improving cybersecurity outcomes, most of the SAT field died years ago—innovation brains eaten away—leaving behind solutions walking around as “compliance checkbox” zombies. This is a huge problem.

At least, it’s a problem if you have much to lose from a successful phishing attack that leads to a data breach or ransomware compromise. If that doesn’t trouble you, feel free to stop reading.

But if you, your executive team, and your board are concerned that data breaches still overwhelmingly start with human errors and compromises, then you should read on.

It’s not just our PoV. Gartner wrote an obituary too.

In 2023, William Candrick, director in Gartner’s Risk and Security Management group, published a presentation entitled “Security Awareness is Dead! Now What?” The first slides deliver a damning broadside against the state of security awareness computer-based training, with two initial assertions. The first is that ten years after Gartner established tracking for this market, 74% of security breaches still start with human error, while 69% of employees still violate security policy. The second is a brutal assessment: “security awareness supports compliance but does little to reduce security risk.”

Ouch.

So, we’re not the only ones with this opinion. You can’t fix something if you don’t acknowledge that it’s broken. But just knowing it’s not working isn’t enough, you need to go a little deeper to find out why it’s broken.

The Separation of Security from Awareness

There are many compliance standards that relate to security or privacy. Compliance is not a bad thing; it sets a standard for minimal effort with some teeth for enforcement. And compliance isn’t automatically divorced from security. In fact, in many cases, security and compliance go hand-in-hand. For example, in PCI-DSS, compliance is directly tied to technical security requirements.

But where security awareness has gone wrong is that compliance has literally nothing to do with direct, technical security requirements or metrics. Instead, compliance is tied to the delivery or completion of training programs: “Did we or did we not do the program and thus avoid legal liability if a breach occurs?”

Infosec teams can’t live or execute against such non-technical deliverables and outcomes, so SAT programs are largely run by HR, training and development teams, or otherwise owned by a legal/compliance department.  In most organizations, the teams that run the SAT program never even talk to security teams.

This reinforces Candrick’s assessment that SAT supports compliance, but not better security.

Click Rates are Security Theater

What happens when a teacher or school’s performance is graded only on the outcome of standardized test scores? Easy—they tilt heavily towards teaching students to pass those tests rather than towards developing well-educated students. And some teachers will be tempted to game things to ensure that students pass.

A similar dynamic affects many SAT programs. Management needs to demonstrate compliance, so they heavily tilt to make that happen. But there’s a critical and absurd difference. With SAT, unlike in schools, the “teachers” (SAT program administrators) also formulate the tests. That means that if the goal of the SAT program is to pass students, they can make that happen.

Let’s make this concrete. If you want to ensure that your SAT “students” pass the test and the program is guaranteed to be compliant, all you need to do is make sure that the bar is low for the simulated phishing emails sent as “tests” to your employees so they pass. Keep the content generic and tests easy, and you get a more “successful” program.

So, when SAT vendors tout successful “click rates,” know that it is a deeply compromised metric, prone to being gamed by program administrators and vendors alike. Think this is outlandish? Let’s consider:

1. Job security for SAT administrators

2. Repeat sales for SAT vendors

3. Human nature

I’ll let you reach your own conclusion.

This is not to say that tracking click rates is useless; these metrics can be used for good. It’s just that, in many cases, click rates are security theater rather than operationally relevant to security outcomes.

The Trap of SAT Cynicism

The fact that SAT has largely sunk into a pit of security irrelevance is unfortunate, but there’s an even more unfortunate result of that irrelevance that’s all too common: SAT cynicism.

Too many people in the industry have concluded that the juice of training people to avoid cyber attacks is barely worth the squeeze.  Some vendors take advantage of this cynicism to play down SAT value and give away free, subpar training products. Others have settled into punting to governance or other vague ways of compensating for underperforming SAT.

But this is a trap. Wise security leaders know that the human element is critical to managing the risk of cybersecurity breaches. But what has eluded most of the industry is a systems-based way of thinking about SAT and email security and how to aim a systematic approach at operationally relevant, measurable cybersecurity outcomes.

How to Unlock SAT Security Relevance: First Build Culture

To put the “security” back in “security awareness training,” you need to start with intention. The compliance gods will not demand a greater sacrifice. You need to demand this of yourself, and this means that there must be tight collaboration between your organization’s security and HR teams.

I think it’s extremely important for us as an SAT and email security vendor to state this clearly: no product nor strategy will make up for a lack of intentional collaboration. If you don’t have that, you won’t achieve anything notable.

SAT and security relevance is a classic example of how culture eats strategy (and technology) for lunch.  We have customers that we’ve seen implement a strong cybersecurity culture, allowing their employees to exercise thoughtfulness, collaboration, strategy, and, yes, training to achieve wonderful outcomes.

Security Relevance Is Based on Outcomes, Not Activities

The textbook definition of SAT is pretty well-known: you train your employees to learn how to spot cyber risks. Then, you run simulations to reinforce and confirm that they have retained this knowledge. As a result, your organization is compliant and more secure. You’ve done your job, right?  The problem is that these are only activities, not security outcomes.

For SAT to be security-relevant in practice, you must define and achieve outcomes that will move the needle. These outcomes need to be ambitious, measurable, and (relatively) easy to understand.

Furthermore, the most relevant SAT outcomes are those that create a direct impact on security operations rather than an indirect or implied impact.

Actionable Intelligence First. Susceptibility Second.

Based on the above principles, the most important metric outcomes are those that lead to greater visibility and actionable intelligence on real threats in your operational email systems.

Primary metric: Number of malicious emails reported. This is the number of phishing attacks detected and reported by employees after bypassing perimeter email security defenses and landing in employee inboxes. 

Related metrics: 

1. Reporting rate (% of employees actively reporting suspicious emails.) This is an indicator of engagement and reporting culture.  Without driving reporting rates, you can’t get actionable intelligence. But the volume of reported emails isn’t enough if the reported emails aren’t of sufficiently high quality.
2. % of reported emails that are malicious. This is a quality indicator of training effectiveness. BTW, you can’t get a high percentage outcome here without training employees on the latest threats that are getting through perimeter defense. If you’re just training on generic, old threats that are already being caught, you’re going to have garbage-in, garbage-out results in terms of reporting.

Secondary metric: Click rate (% of employees that fail simulation tests). This metric is still important to measure employee susceptibility. However, we consider this to be secondary as a metric because it provides implied or indirect impact to security operations, versus direct impact.

Furthermore, the reality is that no matter how well you do at reducing susceptibility, there will always be human risk.  Even well-trained employees can have off days or bad moments. It is impossible to achieve zero percent susceptibility. Perimeter email defenses will always have statistically meaningful leakage of phishing emails. Novel attacks will always appear.

How SAT Becomes a Security Superpower – Scaling through Crowdsourcing

Here’s another reality check: no matter how well-trained your employees are and how much reporting and intelligence generation they drive, there is a finite limit to how much they can catch. After all, your employees have day jobs. But that fact doesn’t decrease the leakage of phishing emails that get through your perimeter defenses. So, how do you fill the gap?

Well, we’ve got good news. Cofense has built the industry’s only crowd-sourced intelligence based on the collective efforts of thousands of other organizations that have trained their employees on real phishing attacks that SEGs miss. Cofense captures that reporting on a global basis and turns it into thousands of monthly IoCs. You can use Cofense SOC tools to automatically remediate these threats. Gap=filled.

Bring SAT Back from the Dead.

If you’re ready to resurrect the security relevance of SAT, look no further than Cofense. We’re the only SAT vendor that integrates the very latest threat intelligence into its training and simulations. We provide incredibly strong foundational education on security awareness, including but not limited to email-mediated threats. Our simulations are all designed by starting from actionable intelligence about what attacks are getting missed and are currently dangerous. We’re also the only SAT vendor that gives you the collective data and tools required to maximize your actionable intelligence so you can catch more phish and defend your organization more effectively. Request a demo today.