Blog

How to Evaluate Your Cybersecurity Analysts Using the MITRE ATT&CK Framework

A Revolution for On-the-Job Cyber Training}

A few years ago, when I was presenting at a cybersecurity conference in Manila, someone in the crowd asked this question: “When I look to hire a cybersecurity firm, what certifications should I look for?”

The polite answer I gave: it depends on what you’re hiring them to do. Cybersecurity is a broad field with an equally wide array of certifications for the different work roles you need to perform.

The more accurate answer ”Knowledge” alone is not sufficient. Certifications provide a knowledge base, but they don’t necessarily equate to on-the-job performance. Organizations need a systematic way to ensure analysts can identify and respond to the latest threat techniques they will likely experience. As cybersecurity grows in importance, this need grows even faster.

A Standard Way of Defining Cyber Jobs

In 2009, recognizing the national challenge of training cybersecurity professionals, the US Government established the National Initiative for Cybersecurity Education (NICE). The NICE framework identifies seven separate cybersecurity career fields shown here:

  1. Analyze
  2. Collect and Operate
  3. Investigate
  4. Operate and Maintain
  5. Oversee and Govern
  6. Protect and Defend
  7. Securely Provision

Each career field contains several work roles with its own list of knowledge, skills, abilities, and tasks (KSATs) that someone would need to succeed in that role. The KSATs also make it possible to develop and standardize training for each role.

For example, consider the Cyber Defense Analyst role, part of the Protect and Defend career field. This role uses a variety of cyber defense tools to analyze events that occur within their environment for the purpose of protecting critical assets from cyber attackers. These analysts sit at the epicenter of attack prevention on your people, processes, and technology. Their KSATs ensure that analysts are proficiently trained to protect and defend. Although these cybersecurity professionals are now trained, how do we know if they can successfully stop a real-world attack?

The best way to ensure proficiency is to evaluate how cyber security analysts perform against actual malicious techniques. To that end, MITRE ATT&CK provides a useful framework for training analysts against the known malicious tactics and techniques they could experience in their jobs.

A Standard Threat-Based Framework

MITRE ATT&CK maps real-world malicious cyber techniques to particular stages of an attack—from initial access to impact. For a Cyber Defense Analyst tasked with protecting an organization from cyber-attacks, the ATT&CK framework is an invaluable reference for contextualizing the types of malicious activity they need to identify and stop.

The ability to train analysts to protect against attack vectors the company faces has proven to greatly enhance a company’s ability to detect, protect, and respond to attacks. Unfortunately, there is a disconnect between Cyber Defense Analyst certifications, job training frameworks like NICE, and the ATT&CK framework used to categorize threats. As a result, organizations have not been able to accurately gauge the competency and proficiency of their Cyber Defense Analysts and teams. This often leads to one of two results: overconfidence in analyst performance and to underestimation of cyber risk; or, uncertainty about analyst performance, leading to overspending on extra security tools and training.

The Big Idea: Use MITRE ATT&CK to Measure NICE Competency

Is there a way to standardize the evaluation of Cyber Defense Analyst performance?

The answer is yes. MITRE ATT&CK provides a useful framework for developing, testing, and improving the knowledge, skills, and abilities of Cyber Defense Analysts. By mapping relevant parts of the Cyber Defense Analyst job role to MITRE ATT&CK techniques, analysts are able to demonstrate competency for many NICE framework requirements by practicing against real-world threats throughout the MITRE ATT&CK framework.

A few Cyber Defense Analyst examples include:

Ability to apply techniques for detecting host and network-based intrusions using intrusion detection technologies.

NICE Knowledge, Skill, Ability, Task (KSAT) Description

Ability to apply techniques for detecting host and network-based intrusions using intrusion detection technologies.

Knowledge of common attack vectors on the network layer.

MITRE ATT&CK Techniques

External Remote Services (T1133)
Drive-by compromise (T1189)

Application Layer Protocol (T1071)
Phishing (T1566)

Using ATT&CK techniques to validate the performance of cybersecurity analysts as part of a broader, structured professional development curriculum like NICE is a great step towards ensuring the workforce is capable of meeting the dynamic threats they face. By synthesizing training standards with threat techniques to test their personnel, organizations can know with certainty that their analysts are competent and ready to perform. For the Cyber Defense Analyst, we can define competency as demonstrated performance against techniques across ATT&CK.

Conclusion

Ensuring Cyber Defense Analysts are ready to prevent, detect, and respond to the latest threats is a national imperative. Training standards, frameworks, and certifications are important, but they mean little if analysts are not proficient against real-world threats they might actually experience on the job. To fully understand and address cyber risk, CISOs, CROs, and other cyber leaders need to understand how their analysts perform against these specific threats.

Synthesizing MITRE ATT&CK and NICE for cyber security evaluations enables organizations to know their analysts have both the knowledge and demonstrated competency to defeat the threats they will face in their jobs. There are now material solutions that enable organizations to know how well, or poorly, their Cyber Defense Analyst performs against the threats you face. To truly train how and where they fight every day.

Special thanks to Ron and Cyndi Gula, from Gula Tech Adventures, and Dr. John Brickey, from Mastercard, for their valuable inputs and guidance to this paper and our overall effort.