Testing: Building a Test Plan – Step 2


The purpose of business risk analysis during software testing is to identify high-risk application components that must be tested more thoroughly, and to identify error-prone components within specific applications, which must be tested more rigorously. The results of the analysis can be used to determine the testing objectives (see Step III) during test planning.

The key to performing a successful risk analysis is to formalize the process. An informal approach to risk analysis methods leads to an ineffective analysis process.

There are four main ‘risk analysis’ methods which testers can employ. Each method applies a different level of formality to the risk assessment procedure.

The first is Judgment and Instinct. This is the most common approach to performing risk assessment during testing. It is an informal approach using the tester’s Knowledge, and experience with past projects, to estimate the amount of testing required for the current project. This can be an effective method, but the fact that the tester’s experience is not readily transferable to other testers is its primary fault. In terms of the SEI Maturity model, this risk assessment process would be somewhere between levels 1 and 2. It is a repeatable approach, but is not formally written down for others to use.

SEI Maturity Model

Premise: The quality of a software system is largely governed by the quality of the process used to develop and maintain the software. Basics: The first step in improving the existing situation is to get management buy-in and management action to clean up the software management processes (walk the talk, as TQMers frequently say). Integration: The second step is to get everyone working together as a team. Measurement: The third step is to establish objective ways of understanding status and predict where things are going in your process. Continuous improvement: Understand that this is building a foundation for continually getting better.

The second method is Dollar Estimation. This approach quantifies business risk using dollars as its unit of measure. It requires a great deal of precision and is difficult to use because results are based on estimates of frequency of occurrence and the loss per occurrence. Thus, the precision is not always there.

The third approach is Identifying and Weighting Risk Attributes. This approach identifies the attributes that that cause a risk to occur. The relationship of the attributes to the risk is determined by weighting. The tester uses weighted numerical scores to determine what the high risk areas are. The scores can be used to decide what application components should be tested more thoroughly than others, and total weighted scores can be used to compare one application to another.

The fourth approach is The Software Risk Assessment Package. This is an automated approach that involves purchasing an assessment package from a vendor. The advantage is the ease of use and the ability to do What-If-Analysis with risk characteristics. Automated risk assessment software packages exist which use the second and third approaches above, however, testers can create their own risk assessment questionnaires with MS Word and do the what-if-analysis with MS Excel.

Any of these approaches could be used for testing projects, however, the third approach, Identifying and Weighting Risk Factors, is the most practical and applies a reasonable level of formality. It also produces the most accurate estimate of risk. This approach identifies and weights risk factors along three project dimensions.

Risk Dimensions

The risk dimensions are identified as Project Structure, Project Size, and Experience with Technology.

With respect to project structure, the more structured a project the less the risks. Thus, software development projects that employ some type of project management / development life cycle approach should be at less risk.

Project size is directly proportional to risk. The larger the project in terms of cost, staff, time, number of functional areas involved, etc., the greater the risk.

Technology experience is inversely proportional to project risk. The more experience the development team has with the hardware, the operation system, the database, the network, and the development language the less the risk.

Task II.I – Assertaining the risk Score

This involves conducting a risk assessment review with the development team. During the session a risk assessment questionnaire is used to structure the process. Each question is asked of the development team members in group session and a consensus is reached as to the perceived level of risk (an alternative approach would be to have individual developers each fill out the questionnaire and consolidate the responses). The questions are closed-end questions with the possible responses of Low, Medium, High, and Not Applicable. A numeric rating is assigned to the response. A final score is calculated using weighting by which the numeric rating is multiplied. The weighted scores are used to identify error-prone areas of the application and to compare the application with other applications.

Output: As the risk assessment review is completed, preliminary score for each question should be recorded on the questionnaires. After the review session, the scores should be converted to weighted scores and entered into a spreadsheet for subsequent analysis. The original questionnaires should be filed away and kept for future reference.

Task II.II – Creating the risk profile

A weighting system is used to compute a score that reflects the importance of each area. Areas which are twice as important can be weighted with a value of two (e.g. an area with medium risk (2 points) could be considered two times as important as the other areas and have a final weighted score of 2 * 2 = 4 points (the weight times the risk points)). A total score is computed for the project, but the individual scores are used to develop the risk profile. Use the following approach to construct the profile.

Once the risk data have been collected, create an MS Excel spreadsheet that computes the weighted scores. Sort the tabulated scores from the highest to the lowest (a pseudo-frequency distribution of sorts) and perform a Pareto Analysis (the 80/20 rule) to determine what project areas fall into the upper 20% of the distribution. These are the areas that are at risk the most and they are the areas that will need to be tested the most. The results are rather obvious when charted using Kiviat Charts (radar charts). The high risk areas standout visually. (The charting should be done based on data sorted in ascending order by question number not on data sorted in descending order by highest to lowest for the Pareto Analysis.)

Output: The results in the sample Kiviat Chart below were taken from the General Risk Assessment Questionnaire data. They are easy to interpret. The further out along the axis a risk factor is placed the greater the risk. In the sample data it is obvious that the information of software development and the lack of documentation are major risk areas.

A second method for interpreting the results is to sort the data from highest risk to lowest risk. This creates a frequency distribution of risk factors also know as a Pareto Distribution. The sample below uses the same data as in the Kiviat Chart but is presented as a Pareto distribution.

n Kiviat Chart Illustrating Sample Risk Analysis Results

n Column Chart Illustrating Pareto Distribution Of Risk Factors

Task II.III – Modify the risk characteristics (Optional)

Once the areas at risk have been identified, a proactive approach can reduce risk. It is suggested that steps be taken to change the development approach or the project structure in order to reduce risk. When these alternatives are not feasible, the process of using the risk information to decide what areas to test becomes even more critical.

Task II.IV – Allocate the resources

Allocate the most test resources to the high-risk areas, allocating less testing resources to medium risk areas and minimal testing resources to low-risk testing areas. A sound strategy is to assure that all, or as many as possible, of the medium to high-risk areas are tested with the scope of the allotted testing resources. A possible split could be to commit 80% of the testing resources to medium and high-risk areas, and commit the remaining 20% to low risk areas. This is again applying the 80/20 rule.

Output: A Risk-Based Test Resource Allocation Plan – is a document containing recommendations for allocating test resources based on the risk analysis results. It should list the risk areas from highest to lowest risk and should assign a percentage of the available testing resources to each risk sector.

Task II.V – Compile a risk assessment database

A risk assessment database has two important functions. First, it can be used to improve the risk assessment process itself. Second, the data can be used to help management plan and structure development projects.

The Risk Assessment Review Session

The entire test team along with end user representatives should be included in the session. It is recommended to conduct the session early in the test process. It should last no more than two hours and should be run formally by the test team manager who facilitates the session. The session should have two objectives:

1) To answer each question on the risk assessment questionnaire.

2) To brainstorm and let the participants voice their concerns about the system under development.

Output: Risk Database – A risk database should be maintained that contains the project name under test, the risk assessment results from the project on a question by question basis, the analysis of the results on a question by question basis, graphs/charts of the results, and the test resource allocation recommendations.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: