Building a Modern CSOC/IRT – Part I: Governance, Roles & Responsibilities and Accountability

A few days ago, LinkedIn’s Ethical Hackers Academy page posted an interesting Infographic about the differences between the Legacy SOC and the Modern SOC.

It deals with issues ranging from the overall philosophy of the entity down to how the cursory should become the core.

But hey, it’s still an infographic; it needs context, it needs a discussion.

As the Director of Cyber Security Operations at IAI, I aimed to achieve just that.

And I think I succeeded, or at least came very close.

It’s part of the philosophy of constantly learning and adapting.

At the time, the SOC was staffed by excellent people passionate about learning and advancing themselves professionally.

But it wasn’t 24/7. And more complex cases were handed off to the Architecture and Technologies Team (where I was under the CISO) and/or to outside contractors on Retainer.

Where I was concerned, I made it a personal point to involve the SOC in any aspects of the Investigations.

To the point where my then-direct manager dressed me down for not including my name in a Key Investigation because I opted to, instead, give credit to all the SOC Analysts I involved in the case.

In a meeting held by the CISO, we discussed the issue of creating a separate IR Team within the Cyber Directorate (CISO) under the management of the Head of Research and PT.

After reading a lot on the current procedures, methodologies, and objectives of SOCs today, I attempted to express what I believed to be the right model the SOC should follow for the first time during that meeting.

A modern CSOC (Cybersecurity Operations Center) or IRT (Incident Response Team) is integral to any organization’s cybersecurity setup. Its primary function is real-time monitoring, detection, response, and mitigation of security incidents and threats.

Why did I set out to Rebuild the SOC?

After consulting with me in January 2020, the CISO appointed me to take over the management of the SOC.

While the existing SOC Manager would go to the Technologies team, where he held more interest.

And since he wasn’t, in essence, a Cyber Security professional, the role exchange between us consisted only of the established process on how to hire new analysts from different HR Subcontractors.

This raised a major Red Flag for Me.

So, I turned to my then-best friend in the SOC – a Shift supervisor—and asked her to show me all the Procedures and Processes the SOC executed.

There weren’t any. There were Play Books and step-by-step tutorials on how to handle specific types of attacks.

So, while my boss’s direction was to establish the Best 24/7 SOC in Israel, I added my own goal: leave for my successor a working entity, with a complete set of well-defined and Documented Procedures and Processes.

How did I go about it?

In my training, I am an officer (reserves) in the IDF.

An Operations Officer. So I set out to define the top-level directive for What the company’s Cyber Security Center is in the proper military structure.

General:

The CSOC/IRT is the Central Nervous System of the CISO Organization within the Company.

The Center is Tasked with the Management and Implementation of the Monitoring, Control, Response, and Return to Competence policies of the entirety of the Company’s resources under the CISO’s Protection mandate.

Operational Concept:

The Cyber Security Operations Center/Incident Response Team is on a 24/7 War of Attrition against attackers of varying skill, opportunity, and motivation levels.

The Incident Response process is an inherent part of the CSOC’s daily operations.

Goal

• Define the CSOC/IRT Roles, Authorities, and Responsibilities.
• Define the day-to-day and shift-to-shift operations of the CSOC/IRT
• Define Goals, Framework, and Metrics
• Including Skills Progress Plan and Personnel Rotation Expectations

Here are my fundamental assumptions:

A Cyber Security Operations Center is not a Career!

An average stint of 2–3 years is the goal.

The core and primary task of the SOC Analyst is Responding to an Incident.

This was at the core of the aforementioned discussion or even argument.

At the end of the Day, even when dealing with your standard Phishing Campaign, a Legacy Tier 1 Analyst is supposed – in my view – to understand the Incident, gather all available Information (triage), expand and find all other Recipients and even Variations on the Theme and Remediate the situation by deleting and adding IOCs where relevant.

Strive and Drive for Proactive, threat-driven cyber resilience.

Let’s Dive a Little:

Personnel:

• I’ve already discussed Bloom’s Taxonomy as applied to Cyber Security Professional Progression.
• In the above chart, Four primary Expertise are mentioned: Penetration Tester, with a prominent role as Threat Hunter for the CompanyDigital Forensics Investigator, again, as Threat Hunter for the CompanyCyber Threat Intelligence.
• Shift Manager
• Managerial role with absolute Authority and Responsibility.
• Other Expertise from the CSOC includes Risk Management, Project Management, Automation and System Development and Integration, SIEM/EDR/SOAR Engineering, and more.
• Work Conditions
• As the Analysts gain experience and expertise and prove themselves in the field, they get more room to pick their Shifts – do fewer Graveyard shifts or other uncomfortable ones.
• Of course, higher pay tariffs are accounted for as well.

Roles:

As stated before, the Roles and Responsibilities – or even the basic definition of what we are looking for – were not well defined (or at all) for the SOC.

So, the entire 5th article of the Cyber Security Center Top Level Directive defines the people’s Roles, Responsibilities, and Accountability.

Director of Security Operations

The director’s Decisions and Instructions are mandatory for all Center Personnel.

Deputy Director/XO

Define and Set the Disciplinary policy.

Shift Manager (changed from Supervisor)

The morning shift manager is responsible for prioritizing a Daily work plan for

• Maintain the complete Cyber Situational Awareness image – events and incidents—during the Shift.

• Prioritize Subject Matter Experts Ongoing Work plans and tasks and engage them in relevant IR Tasks.

Subject Matter Experts

Digital Forensics/Threat Hunt

• Penetration Tester/Threat Hunter Execute tasks assigned by the Shift Manager. Implementing Penetration Testing Tools provides attackers with insight during Incident Response events.

• Complete Penetration Testing Reports and Review External PT Reports.

Cyber Threat Intelligence Officer

Build Attacker Profiles during Incident Response events.

Analyst

• Will see it as their Responsibility and Role to raise Issues and Professional Questions for discussion among the other Analysts and Shift Managers and beyond – to raise engagement, understanding, and training in the relevant field.
• The idea behind the Specific Phraseology, the careful wording, is that each word and sentence define Requirements for Training, Skills, and Tools to be later implemented, to fulfill the goals of the Top Level Directive successfully, and through it, the CSOC/IRT’s overall Strategy.

The Art of Writing Procedures and Directives.

I trained my SMEs and Shift Managers to do this – when you write a Procedure or a Directive, make sure to take special care with the Phraseology.

Don’t be afraid to create Requirements that might require the purchase of new tools or more people. But do make sure you understand what you are asking for, and why.

Of course, the Top Level Directive is subject to changes. As we evolve, as the needs change, so should the Center, and the set of Directives and Procedures must reflect that change.

Additionally, each such Article should be further Detailed in its own Derivative Written Directive/Procedure.

In addition to the Top Level Directive, I have written a separate Detailed Directive for the Roles and Responsibilities of the Shift Manager, the Analyst, and the XO, taking the Phraseology written above and expanding them into their own fully fleshed-out documents.

I asked my SMEs to do the same for their respective areas – Digital Forensics Threat Hunter, PT Threat Hunter, and CTI Officer.

With that, the Core of the CSOC/IRT Directives was documented and ready to be used constantly.

Why the insistence on all this Documentation?

This is the place of Governance.

If you do not set the Vision, you will not achieve it.

You will not get ahead if you do not set the Strategy, the Methodology, and clear Responsibilities.

If you do not set expectations, don’t expect anything to go right.

If you do not define Accountability, then no one is accountable.

These are not just Slogans or Buzzwords.

By defining the Prime Dire… err, I mean, the Top-Level Directive, you create an Anchor from which to Govern the project and entity successfully.

The idea behind the Document is to provide all Team Members, all Employees to Understand the Vision and their roles in it.

And, when shit hits the fan, you base your Lessons Learned processes on a stable anchor.

You are no longer in Limbo, flailing around for corrections, because you compare What Happened to What Should’ve Happened, and then Analyze and Correct accordingly, updating the Procedures and Directives as needed.

Additionally, each Analyst is directed to read all these Directives, and even sign the Analyst’s detailed directive, to set expectations and acknowledge understanding.

CSOC/IRT Core Subdomains

The next step – still Article 5 in the Top-Level Directive – is to define the core Domains or Subdomains of the CSOC/IRT.

I have defined four such subdomains:

1. Automation and Integration, responsible for all tasks and processes that involve the Design, Integration and Implementation of new Systems in the Center.
   In the Professional Vector of Progression above, I mentioned this as one of the fields Analysts can progress and gain expertise in.
2. As defined in 5.4 above, SMEs are the Experts, the Experienced personnel who define new methodologies and disseminate them to the younger, less experienced Analysts.
   Their detailed roles are defined above.
   Their activities, and as a consequence the activities of the other Analysts, constitute the Core Rhythm of the CSOC/IRT.
3. Training and Doctrine, or TRADOC, is the sum of all tasks and processes related to building up the Skills and Talents of all personnel in the CSOC/IRT. Everything from DF, IR, MA, RE, CTI, Automation and Integration and Management skills.
   Again, this was discussed in Bloom’s Taxonomy Article on LinkedIn.
4. The Supervision and Process Improvement subdomain is the general directive that everything should be measured and Reviewed, properly, and Lessons need to be Learned.
   This particular Subdomain, in my view, should be under the direct hands-on jurisdiction of the Center’s Director.
   All personnel must perform AARs and Lessons Learned after each Incident, especially the foibles.
5. But the final review and sign-off, the final Accountability for the Correction and Implementation of Change, Mitigation, and/or Meting Out Penalties must fall on the shoulders of the Director and no one else.

The Core Rhythm

The concept of a Battle Rhythm should be familiar to people with Military Experience.

It is the Scheduling of Daily Operations to allow for synchronization between multiple HQs within a Hierarchical structure.

I found it relevant for the CSOC/IRT, as well.

Both because we had to Sync two additional, smaller SOCs and because it creates a convenient structure for the day and Shift to Shift operations to circle around.

The above image (it’s a screenshot taken from a real working 24h clock done in Excel), combines two principles into one – Pace Layering centered around a 24-hour Clock. This creates a Daily Cycle of Operations.

• The Innermost Layer is the Doctrine – the set of Directives, Policies and Procedures Govern the Center.
• The Training Layer is based on the Doctrine, paced on top of it, aligning the Analysts to the Doctrine via learning.
• Next comes the Static Intelligence Layer – the Threat Landscape, which is Slow to Change overall – we have a relatively static set of Major Threat Actors who try our patience.
• Cyber Threat Intelligence takes the static Threat Landscape layer below it and expounds and expands upon it – producing Actionable Intelligence Outward and slowly updating the Landscape Inward (as mentioned in 5.4.3.4 above).
• Threat Hunting is done by the Analysts and SMEs – to proactively find that which will not be detected by the SIEM or EDR, based on the relevant Cyber Threat Intel.
• Lastly, the Incident Response layer is divided into Three Daily Shifts.

As in the Concept of Pace Layering, the Outermost Layer changes most Rapidly, while the Innermost Layer changes most Slowly, creating a stabilizing force.

An Anchor, to Govern the Center and steady the ship.

In the above sections, I have provided only a taste of the Top Level Directive to define the Vision and Strategy of the Modern CSOC/IRT.

In addition to the Top Level Directive and the other Directives mentioned above, additional Documented Procedures are written as needed. For example, we’ve dealt with a Procedure to Create New Rules (SIEM, EDR, and SOAR), which defined the correct process of Inception, Definition/Design, Implementation, and QA for new Rules.

A Procedure that defined the process of dealing with CTI about companies in our Supply Chain (suppliers or customers) that got hit by major Cyber Attacks. And more.

Of course, the magic doesn’t happen in the documents themselves but in the Implementation and Enforcement of said documentation.

What I haven’t dealt with, here in this article, is the Methodology itself – Chapter 6 of the Top Level Directive.

Next time.