Risks can be positive or negative.
Positive risks are referred to as opportunities and help in business sustainability. For example investing in a New project, Changing business processes, Developing new products. Negative Risks are referred to as threats and recommendations to minimize or eliminate them must be implemented for project success.
In this tutorial, you will learn-
When to implement Risk based Testing
Risk Management Process
Risk Based Testing Approach
Risk Based Testing Approach to the System Test
How to do Risk Based Testing: complete Process
Prioritization and Risk Assessment Matrix
Generic Check list for Risk Based Testing
Risk Based Testing Results Reporting and Metrics
Inherent Risk vs. Residual Risk Assessment
Benefits of Risk Based Testing
When to implement Risk based Testing
Risk based testing can be implemented in
Projects having time, resource, budget constraints, etc. Projects where risk based analysis can be used to detect vulnerabilities to SQL injection attacks. Security Testing in Cloud Computing Environments. New projects with high risk factors like Lack of experience with the technologies used, Lack of business domain knowledge. Incremental and iterative models, etc.
Risk Management Process
Let’s now understand the steps involved in Risk Management Process
Risk Identification
Risk identification can be done through risk workshops, checklists, brainstorming, interviewing, Delphi technique, cause and effect diagrams, lessons learnt from previous projects, root cause analysis, contacting domain experts and subject matter experts. Risk Register is a spreadsheet which has a list of identified risks, potential responses, and root causes. It is used to monitor and track the risks (both threats and opportunities) throughout the life of the project. Risk response strategies can be used to manage positive and negative risks. Risk breakdown structure plays an important role in risk planning. The Risk Breakdown structure would help in identifying the risk prone areas and helps in effective evaluation and risk monitoring over the course of the project. It helps in providing sufficient time and resources for risk management activities. It also helps in categorizing many sources from which the project risks may arise. Risk Breakdown structure sample
Risk Analysis (Includes Quantitative and Qualitative Analysis)
Once the list of potential risks has been identified, the next step is to analyze them and to filter the risk based on the significance. One of the qualitative risk analysis technique is using Risk Matrix (covered in the next section). This technique is used to determine the probability and impact of the risk.
Risk Response planning
Based on the analysis, we can decide if the risks require a response. For example, some risks will require a response in the project plan while some require a response in the project monitoring, and some will not require any response at all. The risk owner is responsible for identifying options to reduce the probability and impact of the assigned risks. Risk mitigation is a risk response method used to lessen the adverse impacts of possible threats. This can be done by eliminating the risks or reducing them to an acceptable level.
Risk Contingency Contingency can be described as a possibility of an uncertain event, but the impact is unknown or unpredictable. A contingency plan is also known as the action plan/back up plans for the worst case scenarios. In other words, it determines what steps could be taken when an unpredictable event materializes.
Risk Monitoring and Control
Risk control and monitor process are used to track the identified risks, monitor residual risks, identify new risks, update the risk register, analyze the reasons for the change, execute risk response plan and monitor risk triggers, etc. Evaluate their effectiveness in reducing risks. This can be achieved by risk reassessments, risk audits, variance and trend analysis, technical performance measurement, status update meetings and retrospective meetings. The table below gives information about the We need to remember that risk increases with changes in technology, the size of the project, length of the project (Longer project timeframe), the number of sponsoring agencies, project estimates, efforts, and a shortage of appropriate skills.
Risk Based Testing Approach
Analyze the requirements. Documents (SRS, FRS, Usecases) are reviewed. This activity is done to find and eliminate errors & ambiguities. Requirements sign-off’s is one of the risk-reduction technique for avoiding the introduction of late changes into the projects. Any changes to requirements after the document are baselined would involve a change control process and subsequent approvals. Assess the risks by calculating the likelihood and impact each requirement could have on the project taking the defined criteria’s like cost, schedule, resources, scope, technical performance safety, reliability, complexity, etc. into consideration. Identify the probability of failure and high-risk areas. This can be done using risk assessment matrix. Use a risk register to list the set of identified risks. Update, monitor and track the risks periodically at regular intervals. Risk profiling needs to be done at this stage to understand the risk capacity and risk tolerance levels. Prioritize the requirements based on the rating. Risk-based test process is defined Highly critical and medium risks can be considered for mitigation planning, implementation, progress monitoring. Low risks can be considered on a watch list. Risk data quality assessment is done to analyze the quality of the data. Plan and define test according to the rating Apply appropriate testing approach and test design techniques to design the test cases in a way that the highest risks items are tested first. High-risk items can be tested by the resource with good domain knowledge experience. Different test design techniques can be used for e.g. using the decision table technique on high-risk test items and using ‘only’ equivalence partitioning for low-risk test items. Test cases are also designed to cover multiple functionalities and end to end business scenarios. Prepare test data and test conditions and test bed. Review the Test plans, Test Strategy, Test cases, Test reports or any other document created by the testing team. Peer review is an important step in defect identification and risk reduction. Perform dry runs and quality checks on the results Test cases are executed according to the priority of the risk item. Maintain traceability between risk items, tests that cover them, results of those tests, and defects found during testing. All testing strategies executed properly will reduce quality risks. Risk-based testing can be used at every level of testing, e.g. component, integration, system, and acceptance testing At the system level, we need to focus on what is most important in the application. This can be determined by looking at the visibility of functions, at frequency of use and at the possible cost of failure. Evaluation of exit criteria. All high-risk areas fully tested, with only minor residual risks left outstanding. Risk-based Test Results reporting and metrics analysis. Reassess existing risk events and new risk events based on Key Risk Indicators. Risk register updation. Contingency plans- This works as a fallback plan/emergency plans for the high exposure risks. Defect analysis and defect prevention to eliminate the defects. Retesting and Regression testing to validate the defect fixes based on pre-calculated risk analysis and high-risk areas should be most intensively covered.
Risk-based automation testing(if feasible) Residual Risk calculation Risk Monitoring and Control Exit Criteria or completion criteria can be used for different risk levels. All key risks have been addressed with appropriate actions or contingency plans. Risk exposure is at or below the level agreed to as acceptable for the project. Risk profiling reassessment and customer feedback.
Risk Based Testing Approach to the System Test
Technical System Test –This is referred to as environment test and integration test. Environment test includes testing in development, testing, and the production environment. Functional System Test– Testing of all functionalities, features, programs, modules. The purpose of this test is to evaluate if the system meets its specified requirements. Non-functional System Test-Testing the non-functional requirements performance, load tests, stress-test, configuration tests, Security tests, backup and recovery procedures and documentation (system, operation and installation documentation).
Diagram below gives a clear overview of the above-mentioned process
System Testing includes both Functional tests as well as Non-Functional tests. Functional testing ensures that the product/application meets customer and business requirements. On the other hand, non-functional testing is done to verify if the product stands up to customer’s expectations in terms of quality, reliability usability, performance, compatibility, etc.
How to do Risk Based Testing: Complete Process
This section covers, Risk based Test Process
Risk Identification Risk Analysis Risk Response Test Scoping Test Process definition
In this process, the risks are identified and categorized, a draft register of risks are prepared, risk sorting is done to identify the significant risks. Risk response involves formulating the test objectives from the risks and selecting appropriate techniques to demonstrate the test activity /test technique to meet the test objectives. Document dependencies, requirements, cost, time required for Software testing, etc. are considered to calculate the test effectiveness score. Test scoping is a review activity that requires the participation of all stakeholders and technical staff. It is important to adhere to the agreed scope of risks. These risks need to be addressed by testing, and all members agree with the responsibilities assigned to them and budget allocated for these activities. After the scope of testing has been finalized the test objectives, assumptions, dependencies for each test stages has to be compiled in the standard format.
Lets, consider the functional requirements F1, F2 ,F3 and Non-functional requirements N1 & N2 F1-Functional Requirement, R1-Risk Associated with F1
Test Objective 1- Demonstrate using a Test that the expected features and functionalities of the system work fine, and the risk R1 can be addressed by functional testing Test-Browser Page testing is done to execute important user tasks and verify that the R1 ( Risk associated with F1) could be addressed in a range of scenarios.
F2-Functional Requirement, R2-Risk Associated with F2
Test Objective 2- Demonstrate using a Test that the expected features and functionalities of the system works fine, and the risk R2 can be addressed by functional testing Test-Browser Page testing is done to execute important user tasks and verify that the R2 could be addressed in a range of scenarios
F3-Functional Requirement, R3-Risk Associated with F3
Test Objective 3- Demonstrate using a Test that the expected features and functionalities of the system works fine, and the risk R3 can be addressed by functional testing Test-Browser Page testing is done to execute important user tasks and verify that R3 could be addressed in a range of scenarios
N1- Non -Functional Requirement, NR1-Risk Associated with N1
Test Objective N1-Demonstrate using a Test that the operational characteristics of the system works fine and the risk NR1 can be addressed by non-functional testing Test-Usability testing is a technique used to assess how easy user interfaces are to use and verify that the NR1 could be addressed by usability testing
N2- Non -Functional Requirement, NR2-Risk Associated with N2
Test Objective N.2- Demonstrate using a Test that the operational characteristics of the system works fine, and the risk NR2 can be addressed by non-functional testing Test-Security testing is a technique used to check whether the application secured or is it vulnerable to attacks, whether there is any information leakage and verifies that NR2 could be addressed by security testing.
Specific Test objectives: The risks and test objectives listed are specific to the test types.
Procedure to design the risk based test process
Prepare a risk register.This records the risks derived from generic risk list, existing checklist, brainstorming session. Include the risks associated with the system functional and non-functional requirements (Usability, security,performance) Each risk is allocated a unique identifier
The Probability(1 Low -5 High ) and consequences(1 Low -5 High ) of each risk are assessed
Test exposure is computed The tester analyzes each risk and evaluates whether the risk is testable or not Test objectives are defined for the testable risks Tester specifies the test activity that should be carried out in a planned way to meet the test objective(Static reviews, inspections, sytem tests, integration tests, acceptance tests, html validation, localization testing, etc.,) These testing activities can be classified into stages (Component Testing/Unit testing, Integration Testing, System Testing, Acceptance Testing) At times, a risk might be addressed by one or more than one test stage Identify the dependencies and assumptions (Availability of skills, tools, test environments, resources) Test effectiveness is computed. Test effectiveness relates to the confidence level of the tester that the risk will be definitively addressed through testing. Test effectiveness score is a number between one and five.( 5-High Confidence , 1-Low Confidence) Estimate of the effort, the time required, cost to prepare and execute these tests.
Test priority number is calculated. It is the product of probability, consequences, and test effectiveness scores.
125-MaximumàA very serious risk that could be detected with testing 1-Minimum àA very low risk that would not be detected with testing
Based on the test priority number the test importance can be classified as High(Red ), Medium (Yellow) &Low (Green). Highest risk items are tested first. Allocation the test activities to the test stages.Designate the group that will perform testing for each objective in the different test stages(Unit testing, Integration Testing, System Testing, Acceptance Testing) What is in scope and out of scope for testing is decided in the test scoping phase For each stage test objectives, component under test, responsibility,environment,entry criteria,exit criteria,tools,techniques,deliverables are defined.
Generic Test Objectives- These generic objectives are applicable to multiple projects and applications
Component meets the requirement and is ready for use in larger subsystems The risks associated with the specific test types are addressed, and the test objectives are accomplished. Integrated components are correctly assembled. Ensure interface compatibility among the components. The system meets the specified functional and nonfunctional requirements. Product components satisfy end user needs in their intended operating environment Risk management strategy is used to identifying, analyzing, and mitigating risks. The System meets industry regulation requirements The System meets contractual obligations Institutionalization and the achievement of other specific objectives established such as cost, schedule, and quality objectives. System, processes and people meet business requirements
Generic Test objectives can be defined for the different test stages
Component Testing Integration Testing System Testing Acceptance Testing
Let’s consider the system test stage
G4 & G5 demonstrates’s the system meets the functional (F1,F2,F3) and non-functional requirements(N1,N2) . Demonstrate using tests that the expected features and functionalities of the system work fine and the risk associated with F1, F2, F3 can be addressed by functional testing Demonstrate using tests that the operational characteristics of the system work fine and the risk associated with N1, N2 can be addressed by non-functional testing Based on the test priority number the test importance can be classified as High(Red ), Medium (Yellow) &Low (Green).
Prioritization and Risk Assessment Matrix
Risk assessment matrix is the probability impact matrix. It provides the project team with a quick view of the risks and the priority with which each of these risks needs to be addressed. Probability is the measure of the chance for an uncertain event will occur. Exposure in terms of time, proximity and repetition. It is expressed in terms of percentage. This can be classified as Frequent(A), Probable(B), Occasional(C), Remote(D), Improbable(E), Eliminated(F)
Frequent- It is expected to occur several times in most circumstances (91 – 100%) Probable: Likely to occur several times in most circumstances (61 – 90%) Occasional: Might occur sometime (41 – 60%) Remote –Unlikely to occur /could occur sometime ( 11 – 40%) Improbable-May occur in rare and exceptional circumstances (0 -10%) Eliminate-Impossible to occur (0%)
Severity is the degree of impact of damage or loss caused due to the uncertain event. Scored 1 to 4 and can be classified as Catastrophic=1, Critical=2, Marginal=3, Negligible=4
Catastrophic – Harsh Consequences that make the project completely unproductive and could even lead to project shutdown. This must be a top priority during risk management. Critical– Large consequences which can lead to a great amount of loss. Project is severely threatened. Marginal – Short term damage still reversible through restoration activities. Negligible– Little or minimal damage or loss. This can be monitored and managed by routine procedures.
The priority is classified into four categories, which is mapped against the severity and probability of the risk as shown in below image.
Serious High Medium Low
Serious: The risks that fall in this category are marked in Amber color. The activity must be stopped, and immediate action must be taken to isolate the risk. Effective controls must be identified and implemented. Further, the activity must not proceed unless the risk is reduced to a low or medium level. High: The risks that fall in this category are marked in Red color ate action or risk management strategies. Immediate action must be taken to isolate, eliminate, substitute the risk and to implement effective risk controls. If these issues cannot be resolved immediately, strict timelines must be defined to resolve these issues. Medium: The risks that fall in this category are marked in Yellow color. Reasonable and practical steps must be taken to minimize the risks. Low: The risks that fall in this category are marked in green color) marked can be ignored as they usually do not pose any significant problem. Periodical review is a must to ensure the controls remain effective
Generic Check list for Risk Based Testing
Comprehensive list of important points to be considered in Risk based testing
Important functionalities in the project. User visible functionality in the project The functionality having the largest safety impact Functionalities that have largest financial impact on users Highly Complex areas of source code and error prone codes Features or functions that can be tested early in the development cycle. Features or functionalities were added to the product design in the last minute. Critical factors of similar/related previous projects that caused problems/issues. Prime factors or issues of Similar/related projects that that had a huge impact on the operation and maintenance expenses. Poor requirements that lead to poor designs and tests that could have an impact on the project goals and deliverables. In the worst case scenario, a product may be so defective that it cannot be reworked and must be completely scrapped this would cause serious harm to the company’s reputation. Identify what kind of problems are crucial to the product objectives. Situations or problems that would cause persistent customer service complaints. End to end Tests could easily focus on the multiple functionalities of the system. Optimal set of tests that can maximize the risk coverage Which tests will have the best high-risk-coverage to time-required ratio?
Risk Based Testing Results Reporting and Metrics
Test report preparation Reporting test status is about effectively communicating the test results to the project stakeholders. And to give a clear understanding and to show the comparison of test results with test objectives.
Number of test cases planned vs. executed Number of test cases passed/failed Number of defects identified and their Status & Severity Number of defects and their status Number of critical defects- still open Environment downtimes – if any Showstoppers – if any Test Summary Report, Test coverage report
Metrics Preparation Metrics is a combination of two or more measures used to compare software processes, projects, and products.
Effort and Schedule variation Test Case Preparation Productivity Test Design Coverage Test case execution productivity Risk Identification Efficiency % Risk Mitigation Efficiency % Test Effectiveness % Test Execution Coverage Test Execution productivity Defect Leakage % Defect detection efficiency Requirement Stability Index Cost of Quality
Analyze the risks in nonfunctional categories (performance, reliability, and usability) based on defect status and a number of test pass/fail status, based on their relationship to risks.
Analyze the risks in functional categories metrics of testing, defect status and test pass/fail status, based on their relationship to risks.
Identify key lead and lag indicators and create early warning indicators
Monitor and report on lead and lag risk indicators (Key Risk Indicators) by analyzing the data patterns, trends, and interdependencies.
Inherent Risk vs. Residual Risk Assessment
Risk identification and analysis should also include inherent risks, residual risks, secondary risks and recurrent risk
Effort and Schedule variation Test Case Preparation Productivity Test Design Coverage Test case execution productivity Risk Identification Efficiency % Risk Mitigation Efficiency % Test Effectiveness % Test Execution Coverage Test Execution productivity Defect Leakage % Defect detection efficiency Requirement Stability Index Cost of Quality
Inherent Risk: The risks that were identified/already present in the system before the controls and responses were implemented. Inherent risks are also known as Gross risks Residual Risk: The risks that are left over after the controls and responses have been implemented. Residual risks are known as the net risks Secondary Risk: The new risk caused by the implementation of risk response plan Recurrent risks: Likelihood that the initial risks will occur.
Test result measurement based on risk helps the organization to know the residual level of quality risk during test execution, and to make smart release decisions. Risk Profiling and Customer Feedback Risk profiling is a process for finding the optimal level of investment risk for the client considering the risk required, risk capacity and risk tolerance.
Risk Required is the level of risk the client needs to take in order to obtain a satisfactory return Risk capacity is the level of financial risk the client can afford to take Risk tolerance is the level of risk which the client would prefer to take
Customer Feedback Gather customer feedback and reviews to improve the business, product, service and experience.
Benefits of Risk Based Testing
The benefits of Risk Based Testing is given below
Improved productivity and cost reduction Improved Market opportunity (Time to market) and On time delivery. Improved service performance Improved quality as all of the critical functions of the application are tested. Gives clear information on test coverage. Using this approach, we know what has/have not been tested. Test effort allocation based on risk assessment is the most efficient and effective way to minimize the residual risk upon release. Test result measurement based on risk analysis enables the organization to identify the residual level of quality risk during test execution, and to make smart release decisions. Optimized testing with highly defined risk evaluation methods. Improved customer satisfaction – Due to customer involvement and good reporting and progress tracking. Early detection of the potential problem areas. Effective preventive measures can be taken to overcome these problems Continuous risk monitoring and assessment throughout the project’s entire lifecycle helps in identification and resolution of risks and address the issues that could endanger the achievement of overall project goals and objectives.
Summary:
In Software Engineering, Risk based testing is the most efficient way to guide the project based on risks. The testing efforts are effectively organized, and level of priority of each risk item is rated. Each risk is then associated with the appropriate test activities, where a single test having more than one risk item, then the test is assigned as the highest risk. Tests are executed according to the risk priority order. Risk monitoring process helps in keeping track of the identified risks, and reducing the impacts of residual risks.
