A few weeks ago, I had a conversation with a fellow CISO about running effective phishing simulations. There’s some debate as to the value of this practice and I wanted to talk about how I’ve been able to get value from these simulations. I’ve been fortunate enough to run some very successful phishing simulations and work with some of the best in the industry to conduct them. Through trial and error, I’ve found a formula that seems to work for me and while I am always learning, I wanted to sketch out a few things that seem to work as well as a few things to absolutely avoid when conducting a phishing simulation.
First, let’s establish why we are doing this. Why conduct a phishing simulation at all? Phishing simulations are an experiment to see how your employees (and most likely a subset of your employees) will react and respond to a phishing message without any harm or consequences. There’s a few specific things we’ll want to learn:
- What information is available for phishers to use to develop realistic pretexts?
- How easy or difficult do we have to make our pretexts to successfully phish our employees?
- Do employees know how and where to report suspicious messages?
- Which groups of people need more training and which groups are performing well when it comes to spotting and reporting phishing messages?
One additional outcome that you’ll want to keep an eye on as the campaign progresses: what are the weird and unpredictable outcomes that occur as a result of the simulation? Phishing simulations are a little bit like fuzz testing humans. You throw weird input at them and you see how they respond. It’s an educational process because there will be outcomes that you never could have predicted that the simulation will bring to light. This is one of the ways that an effective phishing simulation can be incredibly valuable.
As a mentioned previously, when it comes to running phishing simulations, I’ve had the opportunity to work with some of the absolute best in the industry. Rather than spend my money on the care and feeding of my own phishing platform, I’ve spent the money to work with these people instead. Their expertise and experience when it comes to pretext development, measuring performance, and developing recommendations is far more valuable than the actual act of sending phishing messages (which in the course of a phishing simulation makes up very little of the overall level of effort). It’s also been helpful to have an independent perspective on how a security program is tuned to handle phishing reports and how employees are (or are not) reporting phishing. These insights are the most valuable outcomes from any phishing simulation.
Pretext development is not a trivial endeavor; it’s probably the most important component of a phishing simulation. Get your pretexts wrong and your simulation may be tone deaf at best and value destructive at worst. It’s very important to use pretexts that are both realistic but do not use fear, deception, or threats. It’s also important to keep the “Chris Hadnagy rule” in mind, which is to leave people feeling better for having met you. This is easier said than done but the bottom line here is to not make people feel like garbage. Here are some good examples of pretexts that can make people feel like garbage:
- Click here to learn about your holiday bonus.
- Failure to complete the linked form may result in disciplinary action.
Do real phishers play by these rules? Of course they don’t. Real life phishing pretexts could include anything, including elements of the above. Our point is not to simulate the same tactics of the real life phisher but rather to simulate how our colleagues will react to phishing messages. You don’t need to resort to an “anything goes” mindset to achieve this goal. For most people, the phishing simulation in and of itself is not the training moment. The training moment comes after; once you have determined where your colleagues have knowledge gaps.
Who should know that you’re performing a simulation? I’ve struggled with this question. At an absolute minimum, you have to have executive-level sponsorship. You don’t necessarily need to tell them when and what pretexts you will use but you do need to make sure that they understand what you’re doing, why you’re doing it, and what they can expect. Once you’ve done this, you have to decide who needs to know so that you don’t create havoc when you flood your user base with simulated phishing emails. The most recent formula I’ve found (which is working) is to let a few key people in IT know. Make sure to include any mail administrators. As for other groups like user support and Helpdesk, rather than let them know about your plans, make sure they are trained as to how to respond to phishing reports. You can use the simulation as a way to test whether or not they know what to do when a report comes in. This way, it doesn’t matter if the phishing messages are real or a simulation. The response should be the same regardless.
It’s important to keep in mind that despite your best efforts, careful planning, and thoughtful execution, there could be unexpected outcomes that blow up in your face. Unfortunately, it is difficult to plan for these outcomes. My advice is just to be ready to handle any fallout from these unexpected outcomes. Own up to the simulation, talk to people who are impacted, and make it right. I’ve had some very weird, uncomfortable things happen as a result of running phishing simulations. If you lead with accountability and compassion, you should be able to avoid further damage and get back to the business of wrapping up your simulation.
The final thing to keep in mind as you’re planning for and executing your phishing simulation is the idea of punishment. In general, you should not be punishing your employees as the result of your simulation. Whether the attack is a simulation or real, punishing your employees for being victimized by a phishing attack is not conducive to building a strong culture of security. It doesn’t breed goodwill and it does not open doors for security. So repeat after me: I will not punish employees who get phished as part of my simulation. Let’s be honest: you know people are going to click. You know they’re going to send files to the phishers. The point is to understand how this is happening at scale so you can work to minimize risk and contain damage.
Once you’ve got all of your results, it falls into four categories. These are:
- Didn’t click, reported (this is the number we want to maximize)
- Clicked, reported (they caught their own error and fessed up)
- Clicked, didn’t report (the “uh-oh” category)
- Didn’t click, didn’t report (reasons may vary)
Over time you’ll want to measure these numbers, maximizing the number of people who report and minimizing the number of people who click. If your security awareness efforts are paying off, you should see these numbers improve over time (unless you are increasing the difficulty of the pretexts which in that case could actually have the opposite effect).
While the practice of running internal phishing simulations can be controversial I am a believer in the value that this exercise can provide. I’ve seen how these simulations can improve a security awareness program and lead to more effective employee communication. How have you used phishing simulations to improve your human defenses and resiliency to phishing attacks? Let me know in the comments below and thanks for reading.
