Critical Testing; Exposing Your Company’s Security Gaps

It was a standard fire drill, or so they thought until five different secured areas had been breached. All the personnel had evacuated the building, making the fire marshals extremely happy, but it would take much explaining as they now had to self-report security failures to various regulators. All of this could have been avoided had the site security taken seriously the drill after being questioned by my team.

Previously, I provided a high-level overview of how to protect your business. This article will explore the first steps of defining the risks or the CHECK stage of the PLAN-DO-CHECK-ACT(PDCA) lifecycle. Though we previously discussed the CHECK stage regarding validating the plans, we did not discuss the actual exercise and testing of your plans.  

You Need to Be Asking ‘What If?’

Whether it is business continuity (BC), disaster recovery (DR), cyber security (CS), or just traditional facility evacuation drills, like a fire drill, one must test one's plans thoroughly, always asking “what if”. Your team may draft the best plans possible, but the purpose of testing and exercising is to identify any gaps that exist. How can you identify gaps if you do not understand the threats, risks and purpose of testing?

Using the fire drill above as an example, what is the purpose of testing these evacuation plans? Ask any site security team, and they’ll tell you ‘To get all the personnel out as quickly and orderly as possible and to save lives.’ At least, that is what I was told when I observed the quarterly fire drill at my client's facility. However, while I observed the drill, I noticed that all the secured area doors that are normally electronically locked remained open, resulting in these areas being unsecured.  No one from the security operations center (SOC), where I was observing from, had these normally secure areas on the camera.

They did observe the campus evacuation route(s) and the various mustering points. During the drill on one monitor was a real-time report resulting from the mustering badge points, counting and confirming the number of personnel evacuated. The SOC could account for anyone not checking in at a mustering point and produce a list of names to follow up on. Their systems worked, and if this had been an emergency, they could have successfully produced a list of unaccounted-for personnel.

However, in my typical fashion, I began to ask the ‘what ifs.’ I asked the head of security, “Have you ever considered the threat could be a diversion to someone wishing to harm us? We had the secured pipeline control room, the trading floor, the data center, the executive offices, and the research and development areas open. All were left unsecured, and anyone could have walked right in with complete, unobstructed access.”

The head of security said, “That is why we are watching the cameras.”  I told him, “ I observed the team watching the main gates and the mustering points but not any of the secured areas.”

The Purpose

All plans should address how you will respond during a scenario. Unless leadership states otherwise, it is not acceptable for any plan or response to increase the company's risks. This means you must ask the "what ifs." In the real-life example I provided, no one asked how this traditional fire drill could increase the company's risks. It was not even considered when I posed the question.

The Hot Wash

The drill itself took about 30 minutes and afterward, there was a hot wash. A hot wash, sometimes called a postmortem review, is intended to have the stakeholders and participants convene, review the objectives, identify gaps to be addressed after actions, and track them to completion. This is a vital step in testing your plans. If you have no gaps or nothing identified that could improve the response, then you are most likely not adequately testing your plan.

During the hot wash, I brought this concern up and all agreed with the head of security, that it is not a consideration during a fire evacuation or drill. I asked if we could at least “identify this risk was brought up by my team and inquire from leadership guidance on the matter.”  The team leading the hot wash declined my request and simply drafted a report, bragging about how successful the drill went, and the Fire Marshal signed off on it.

Understanding The Gaps

When testing your plans, it is vital to ask not only the ‘what if’ but also to identify objectives that will push the limits of the drafted plan. This may take a team to look at the plan, work collectively on identifying objectives, and propose potential threats or gaps in the plan while doing so. So, when it was time to plan for the next fire drill, my team went to work preparing.

I approached the site security office and expressed my concerns that we should include this as an objective. Again, was shot down. I met with some senior leadership to tacitly share my concerns and obtained their approval to highlight this risk. If they did not share the concern, I did not wish to lose a client and would document the identified risk and ensure they signed off on it. However, they agreed with my concern and approved my team addressing this risk in whatever manner we needed to without the planning team knowing about it.

Fire Can Burn More Than Just Buildings

The fire drill had been scheduled, and my team carefully worked with a team of security agents from other sites. All of whom could be trusted to remain quiet in our planning and had the skill set to pull this off. So, on the day of the drill, they convened with some members of my team and confirmed our objectives.  First to gain entry into the facility without proper identification. Second was to attempt to breach the critical and secure areas and leave some evidence of the breach. Third was to exit the facility without being identified. The last was to provide evidence of, where possible, an ongoing breach.

As the fire alarms began to sound, I stood in the SOC and observed the thousands of personnel exiting in an orderly fashion. I asked again how they would identify anyone if this were a false flag event, but no one took my inquiry seriously. I saw the security officers switch cameras occasionally and I spotted a few of my ‘spy team’ walking against the flow of unchallenged people. No one else in the SOC observed anything besides what they wanted.

Smoldering Flames Continue to Burn

About 20 minutes later, the all-clear alarm and notification went out to the campus and all personnel. All the secured areas, gates, and entry points had been reactivated, forcing each person to badge in as they entered. At this point, red flags began to fly.

About 15 people somehow lost their badges when accessing the facility; they worked in one of the secured areas. Though security began issuing temporary badges, no alert was triggered. In fact, the 15 badges lost happened to be of the same 15 personnel unaccounted for at the mustering points. This should have set off alarms as the drill could not account for all the personnel, which they had repeatedly proven in past drills they could do.

 As staff re-entered the secured areas, they were greeted with a variety of items alerting them that someone had been in the area. Some alerts consisted of signs or tags stating “breached call the SOC to report,” USB drives scattered around, multiple drawers opened, folders (neatly) placed in piles, and every missing badge sitting on the CEO's desk. The SOC’s phones began to ring.

Though it was very apparent to the security manager that my team and I were behind this breach, he quickly realized that he had to investigate it and self-report the matter. I recommended that his team take this very seriously, as my team could not only breach the facility but also leave breadcrumbs. I began to show him the pictures the team took in these sensitive areas.

After security retrieved all the evidence of the breach, or so they thought, they again initiated the hot wash. To say tension was high would be an understatement. I am sure some participants in the meeting would have loved the opportunity to take a shot at me.  We reviewed how the team made entries, stating that they had identified key personnel to attempt to take their badges and walk into each of these areas, and did not even need the heisted badges to do so. I reviewed all the breaches (signs) and shared the selfies as evidence.

I asked the head of security how he would know if this had been an actual breach or if they “had neutralized all the threats.” He stated, "The entire event would be evident from the camera feeds.” It was this statement that I pushed for an after-action item assigned to him for each ‘spy’ to be identified, including how they came in and where they went, and to ensure us there were no further threats.

After-action Items Need To Be Validated

About a week later, this specific after-action item was reported as being completed. I asked to meet with the facility security manager to see the evidence, and we did so with one of the security VPs. He could identify where a few people had entered, going against the flow of evacuating personnel, but none of the images captured showed an identifiable face. Badging records showed that none of the badges stolen had been used to enter the secured areas; in fact, he did confirm that all electronic locks are disabled and badges are not needed to exit a secured area during a fire.

He also had a complete inventory of the evidence from the breached areas and added that they found two other items not identified by the initial search. They consisted of a mocked badge reader skimmer and a packet tap/network sniffer in one of the data centers. At that point, the VP began inquiring about the other six items I had his pre-approval to leave in these key areas: mock recording devices, mock cameras and other network sniffers.

No-Fault Testing

Though my point was well made, and we did need to self-report to the appropriate regulators of the breach, I quickly reminded the facility security manager that one of the objectives and rules of all exercises here was the no-fault testing. No one should have action taken against them for failed items, gaps, and mistakes during an exercise. After all, it is the point of the exercise.  It is vital to communicate this to all participating in your exercises. You do not want someone to be paralyzed by the fear of losing their job and not act.

The VP confirmed that no one was losing their job, and she hoped that all concerns would be considered in future event planning.

Summarizing

The purpose of testing is threefold. First, it aims to identify gaps. If no gaps are found, the team planning the exercise needs to ask whether the objectives have been realistic.

The second is to build ‘muscle memory,’ that is, to familiarize personnel with how to respond, implement plans, document challenges, and the escalation process. Too often, I have witnessed how response teams freeze during a response because they are unfamiliar with incident response and reporting processes.

The last purpose is to improve the quality of response plans and ensure confidence in a company’s ability to address and reduce risk.

The next time you’re planning a ‘traditional drill’ consider asking how might we be exposing our company to risks during this evacuation?

My next article will focus on the last stage of the PDCA lifecycle. Future articles will focus on various program enhancements to address Cyber Resiliency, Disaster Recovery or avoidance and other areas of concern.

James Knox is a resiliency expert with an innovative spirit who thrives when building meaningful solutions to various daily problems in the corporate world. He is an avid outdoorsman and loves extreme rock crawling, fishing, and hunting. As a survivalist, James has learned from necessity how to prepare for life’s bumps and thrive with practical and sensible solutions, supporting his family's self-sustaining lifestyle.