How do you accurately measure the success of your security awareness phishing exercises?
Simulated phishing exercises are a great way to identify current behaviour when staff are presented with social engineering attacks.
They are presented with real-life situations that a cybercriminal might target them with. After this, put the staff through some security awareness training and the problem is solved. Right?
So, how do you measure the success of a phishing simulation? If you are like most people in the cybersecurity community, you may find yourself focusing on statistics like click rates to determine whether you’ve led an effective anti-phishing campaign. After all, if subsequent simulations generate fewer clicks, surely that means your employees have understood the message you are trying to get across about the dangers of phishing, right?
Unfortunately, this simply is not the case. And the reason is click rates and other popular metrics used to evaluate security awareness training does not tell the whole story.
Click rates are easy to control or fake. More difficult to identify simulations, more persuasive ones, get high click rates while simple ones, less persuasive, generate fewer clicks. If someone only wants to improve click rates in phishing simulations, all they must do is start with a difficult simulation and lower the bar over time.
Click rates only measure a single point in time. Each time we measure a click rate, we are looking at different people encountering different simulations, under different situations.
Measuring click rates from month to month or from simulation to simulation might seem logical on a bar chart, but it does not allow us to understand employees’ true learning curve. If the click rate is 30% in January and 20% in February, we might assume we improved, incorrectly so.
We need to look at individual users, how regularly they click, how often they did not, how persuasive were the emails, did they report the phishing email and are they improving after training.
A user clicking multiple times in a row may be a sign of a root cause (such as the requirements of their role, lack of necessary workplace adjustments, a high-stress situation, bad personal or work issues, lack of acceptance of responsibility, or confusion about the current training material) or just bad luck. What about the person who only clicks occasionally, but usually manages to pick the phishing email?
When it comes to helping your organisation avoid phishing attacks, you need science rather than intuition to guide your efforts. Now is the time to abandon your click rate fixation and begin to understand if your phishing awareness efforts are really working or not.
What metrics should you be using in combination to create one complete and accurate picture of whether your training program is meeting its goals or not? How can you determine if your employees are really becoming ever-more successful in avoiding phishing attempts with each iteration?
The changes in click-through rates are different depending upon the inherent persuasiveness of the phishing emails. Spear phishing emails have a higher click rate than generic phishing due to the focused nature of the email.
Therefore, measuring a general/average click rate will not provide you with the correct data to evaluate the success of your phishing program.
The best way to ensure that you accurately measure the success of your security awareness simulated phishing exercises is to not to count the click rate as raw data, but to accurately analyse the susceptibility likelihood and reporting response rate incorporating the persuasiveness index.
Layer 8 Security has developed a newly developed social engineering framework called the “PC-ratio”. This involves the persuasive index of the attack, correlated to the capture rate and the reporting index.
This data is then matched to other user behaviour information and other data. We based the persuasiveness scoring on Cialddini’s six key principles, reciprocity, scarcity, authority, commitment, and consistency, liking and consensus (or social proof).
Persuasiveness
The principle behind it is the persuasiveness of the phishing attack email is how easily persuaded would someone be with the different types of emails.
- These are the common basic Nigerian scams and lost inheritance or massive prize giveaway scams. These should not be terribly hard to identify and most people should not be susceptible to falling for these.
- These are the common scams using Banks, ATO, retails stores, and other common scams. Most people these days are aware of these and under normal circumstances, they do not fall for these.
- These are the most common Spear Phishing emails that we see today. They are coming, supposedly, from within the organisation or trusted companies. Many people fall for these are they believe it is a trusted source.
- (Business Email Compromise) These are very targeted emails coming from an executive within the organisation to request specific actions. These are often very well crafted and hard to identify. Due to their nature, these are quite hard to detect and the person receiving them often feels pressured to respond to these.
Each of these categories has differing weights to reflect their level of persuasiveness and the expected difficulty in detecting them.
Another component of the persuasive index is the position/title of the person being targeted. Differing people and their position within the organisation will respond differently to emails. A CFO will respond differently than a receptionist to a BEC email.
Click Rate
Another measure that is taken into account to calculate the susceptibility likelihood is the Click rate. This is an obvious metrics, but it must be taken into account with the other factors via the algorithm.
Response Index
To understand staff behaviour, we also need to take into account the Response Index. This is the measurement of the staff reporting the suspected phishing exercise, did they just open it, delete it, click the link etc. These numbers allow us to better understand how staff responded to the phishing email and how they may respond in the future.
Repeat Offence
Finally, we need to consider repeat offenders. Are some staff members continuously falling for these simulated attacks, subsequently not reporting them, and placing the organisation at risk?
Calculation
All these factors are then combined using the calculation below to provide us with the Susceptibility Likelihood or more commonly called the Human Risk.
- Susceptibility likelihood (Risk) = R
- Persuasiveness index = P
- Click rate = C
- Response index = I
- Repeat Offence Rate = O
- Meantime between clicks = M
- Training schedule = T
R = (((( P x C ) – I ) (1+O)) – M )/T
All factors have algorithms attached to them so when they are placed within this simple formula, they provide a risk score for each staff member within the organisation, allowing you to continue to measure their success and response to training.
