9.3 Create Risk Evaluation Criteria

During this activity you define your organization's tolerance for risk by creating evaluation criteria. These criteria are measures against which you evaluate the types of impact you described during the previous activity. An organization must explicitly prioritize known risks, because it cannot mitigate all of them. Funding, staff, and schedule constraints limit how many and to what extent risks can be addressed. This activity provides decision makers with additional information that they can use when establishing mitigation priorities.

Step 1: Review Information

You need to review relevant background information to help you define evaluation criteria. Such information includes the following:

• Strategic and/or operational plans that outline the major business objectives of your organization

• Legal requirements, regulations, and standards of due care with which your organization must comply

• Insurance information related to information security and information protection

• Results from other risk management processes used by your organization

You can also use the narrative impact information that you documented during the previous activity. Your goal is to develop an understanding of any existing organizational risk limits based on strategic and operational plans, liability, and insurance-related issues. These data are important in shaping evaluation criteria.

Evaluation criteria are highly contextual. For example, while $1 million may represent a high impact for one organization, it might signify only a medium or low impact for another. Also, some organizations will have risks that could result in a loss of life, but others will not. The contextual nature of evaluation criteria is the reason every organization must define its own criteria and why you need to review relevant background information.

Step 2: Define Evaluation Criteria

In this step you define your organization's evaluation criteria. Discuss the following questions for each area of impact (see previous activity for a discussion of areas of impact):

• What defines a "high" impact on the organization?

• What defines a "medium" impact on the organization?

• What defines a "low" impact on the organization?

You are trying to define specific measures that constitute high, medium, and low risks for your organization in each case. For example, a low impact on productivity might be three lost days, whereas a high impact might be three weeks. As always, make sure that you record this information.

Now let's look at evaluation criteria in the context of an example. The analysis team at MedSite included a member from the risk management department to help them construct evaluation criteria. Prior to the process 7 workshop, the staff member from the risk management department worked with one of the analysis team members to collect background information. They gathered the organization's operational plan and information about legal requirements and regulations.

Prior to the workshop, all members of the team reviewed the information. They selected the following areas of impact for which to create evaluation criteria:

• Reputation/customer confidence

• Life/health of customers

• Productivity

• Fines/legal penalties

• Finances

• Facilities

The team discussed what constitutes a high, medium, and low impact on the organization for each of the relevant areas and recorded the information. Figure 9-2 highlights the evaluation criteria for reputation/customer confidence. You will find a complete set of criteria for the example in Appendix A of this book.

Scenario Planning and Probability

You might have noticed that we are focusing only on impact at this point. A second commonly used risk measure is probability. For information security risks, probability is a more complex and imprecise variable than is normally found in other risk management domains, because risk factors are constantly changing. Probability is highly subjective in the absence of objective data and must be used carefully during risk analysis.

Because objective data for certain types of information security threats (i.e., human actors exploiting known vulnerabilities) are lacking, it is difficult to use a forecasting approach based on probability. Without objective data, it is impossible to develop a reliable forecast of the future [HBR 99]. What you can do, however, is carefully analyze threats to limit the range of potential options, so that you become able to manage your risk. In information security, you can define a range of threats that could affect a critical asset, but you cannot reliably predict which scenario(s) will occur. However, by broadly defining the range of threats that your organization faces, you can make fairly certain that those that develop do so within the defined bounds.

The analysis approach that we are describing here is derived from a technique called scenario planning. A range of threat scenarios, or a threat profile, is constructed for each critical asset. The scenarios in each threat profile represent those in the probable range of outcomes, not necessarily the entire range. Because data with respect to threat probability are limited for the scenarios, they are assumed to be equally likely [Van der Heijden 97]. Thus, priorities are based on the qualitative impact values assigned to the scenarios.

Probability values can be factored into prioritization, but you must take care when doing so. Remember, probability is a forecasting technique based on the premise that you can forecast threat probability with reliable precision. Thus, in many cases you may be forcing decisions based on probability forecasts that are nothing more than guesswork. Nonetheless, incorporating probability into a risk analysis continues to be a popular topic. Section 9.5 considers an approach for incorporating subjective probability in OCTAVE.

When Should You Create Evaluation Criteria?

Note the following two conditions governing risk evaluation criteria:

• There is one set of evaluation criteria for all assets; the criteria are not unique to an asset.

• Evaluation criteria are created for predefined areas of impact, which are related to the organization's key business objectives.

Because evaluation criteria are asset-independent and address broad organizational issues, you could create them earlier in the evaluation process. Some organizations decide to add this activity to process 1, the senior management workshop. By doing so, these organizations are able to gather input from senior managers with a broad perspective on organizational issues. Another idea is to create evaluation criteria when preparing to conduct the OCTAVE Method, as part of your tailoring activities.

If you have previously conducted the OCTAVE Method in your organization, you could use the set of criteria that you already created. If you decide to use evaluation criteria from a previous evaluation, remember to review them and adjust them as appropriate before using them in the current evaluation.

No matter when you create evaluation criteria, it can be a long process. You will probably find that it is also an iterative process. An organization will often revisit its evaluation criteria and adjust them after trying to use them. However, once you are satisfied with your criteria, you have a useful tool for interpreting risk. In the next activity we show how you use this tool.