I’m continuing my long series of posts that describe how to implement an information security program. Currently, we’re in the section called “How to Measure Cyber Risks.”
Last week, I described part 1 of my workflow to collect scores from experts.
At this point, you are ready to ask the expert’s supervisors to help prepare the experts for the interview.
Ask the supervisors to tell their experts ahead of time that the purpose of the scoring is to find opportunities for improvement, not to catch people doing things wrong and get them into trouble. This isn’t an audit; it’s a management review.
Also, mention that the raw data gathered will not be shared with executives; only the summary data. Ask the supervisors to encourage their expert to tell the respectful truth when assigning scores.
To increase the likelihood, the expert will score accurately and truthfully, ask their supervisors not to attend your meeting with their expert. If they insist on coming to the interview, ask what it would take for them to feel comfortable not participating and then try to meet their request.
After giving the supervisors a few days to talk with their experts, send a meeting request to each expert to gather scores.
Budget 15 minutes per expert to set up appointments. Expect to spend up to 2 hours per meeting with experts to gather scores. The primary drivers of the interview time are the number of people and the number of questions.
Every interview is a balancing act: Keeping an eye on the clock, while trying to not rush through it. If you are concerned about conducting the interviews, try practicing with a friend before you meet with your first expert.
Here’s our standard agenda, that assumes a sixty-minute meeting:
At the start of the meeting, using words similar to those you said to their supervisor, take two minutes to explain the purpose of the meeting.
Then, briefly explain how the data will be used after it’s collected.
Next, pass out the supporting materials and give the experts a 5-minute training:
- Show them how the “zero-through-ten” score system works.
- When determining a score, ask them to consider what they have experienced with each outcome in the previous six months and what they expect will happen six months into the future.
- Even if they don’t see how the outcome is delivered, they should be able to choose a scored based on what they have seen delivered.
- For example, they may not be involved with creating or using incident response plans, but they probably have seen how well they work.
Note this approach emphasizes learning to “score as you go.” Experts may feel awkward at first using this approach. Your job during this early phase is to minimize your own talking and give them time to figure it out.
Finally, briskly step through the questions one-by-one and collect scores over the rest of the available time.
- Read each question out loud while the expert follows along.
- After each question is read, allow silence to enter the room to give the expert a chance to consider the question and use the score key.
- If the expert asks for clarifications, do your best to help but let the expert assign the score.
- Document all scores in your spreadsheet as you go, including any insightful comments made by the expert.
Let’s look at a few time estimates for gathering scores from experts. These estimates do not include the setup effort.
- Using the “Good Data” quality method, a Cyber Risk Opportunities employee spent 46 hours over 4 weeks collecting data online from over 150 experts across the world at a $1 billion-dollar financial services company.
- Using the “Better Data” quality method, the Chief Information Security Officer at a $4 billion-dollar county government spent 29 hours over 5 weeks interviewing 22 experts.
- Using the “Better Data” quality method, a Cyber Risk Opportunities employee spent 22 hours over 2 weeks at a $300 million privately held company interviewing experts from 10 departments. Interviews were face-to-face either in person (60%) or over video call (40%). They were scheduled back-to-back in 4-hour blocks with a short break for the interviewer somewhere near the mid-point of each time block.
Next week, I’ll describe how to generate scores from your cybersecurity systems, which can increase the overall quality of your scores.
Cyber Risk Opportunities provides middle market companies with cost-effective Cyber Risk Managed Programs to prioritize and reduce your top cyber risks, including the specific requirements of GDPR, PCI, HIPAA, SOC2, ISO 27001, DFARS, and more.