Activity: Evaluate Test
Purpose
|
|
| Steps | |
| Input Artifacts: | Resulting Artifacts: |
| Worker: Test Designer | |
| Workflow Details: |
Analyze Test Results
and Submit Change Requests 
Purpose
|
| Tool
Mentors:
Rational LogViewerTM: Rational TestFactoryTM: Rational PurifyTM: Rational QuantifyTM: Rational PureCoverageTM: Rational ClearQuestTM: Rational LoadTestTM:
|
In the Test Execution activity, the results of test were reviewed to ensure that the testing executed to completion, and the reported test results were not influenced by non-target-of-test factors.
In this activity, the test results analyzed to identify the details regarding the differences between the expected test results and the actual test results. Differences indicate potential defects in the target-of-test and should be entered into a tracking system as change requests, and the appropriate next corrective actions taken.
Review each reported test failure and identify the necessary information needed to enter a change request into the appropriate tracking system. The information entered should be accurate, appropriate, and meet accepted change request tracking standards or guidelines.
Evaluate Requirements-based Test
Coverage
Purpose
|
| Tool
Mentors:
Rational TestManager: Rational TestFactory: |
To evaluate requirements-based test coverage, you need to review the test results and determine:
- The ratio between how many requirement-based tests (test cases) have been performed in this iteration and a total number of tests for the target for test.
- The ratio of successfully performed test cases.
The objective is to ensure that 100 % of the requirements-based tests targeted for this iteration have been executed successfully. If this is not possible or feasible, a different test coverage criteria should be identified, based upon:
- Risk or priority
- Acceptable coverage percent
See "Key Measures of Testing" in Introduction to Test.
Document the results in a Test Evaluation Report for this iteration.
Evaluate Code-based Test Coverage
Purpose
|
| Tool
Mentors:
Rational TestFactory:
Rational PureCoverage: |
To evaluate code-based test coverage, you need to review the test results and determine:
- The ratio between the code that has been executed during test (such as lines or statements) in this iteration and the total code in the test target.
The objective is to ensure that 100 % of the code targeted for this iteration phase been executed successfully. If this is not possible or feasible, a different test coverage criteria should be identified, based upon:
- Risk or priority
- Acceptable coverage percent
See "Concepts: Key Measures of Test".
Document the results in a Test Evaluation Report for this iteration.
Analyze Defects
Purpose
|
| Tool Mentor |
To analyze defects, you need to review and analyze the measures chosen as part of your defect analysis strategy. The most common defect measures used include the following different measures (often displayed in the form of a graph):
- Defect Density - the number of defects are shown as a function of one or two defect attributes (such as status or severity).
- Defect Trend - the defect count is shown as a function over time.
- Defect Aging - a special defect density report in which the defect counts are shown as a function of the length of time a defect remained in a given status (open, new, waiting-for-test, etc.)
Compare the measures from this iteration to the results from the analysis of previous iterations, to get a view of defect trends.
It is advisable to show the results in diagram form.
See "Concepts: Key Measures of Testing".
Determine
if Test Completion and Success Criteria Have Been Achieved
Purpose
|
Review the test strategy defined in the test plan. It should present test criteria in terms of test coverage and, or defect evaluation. Examine the test results, defects, and the analysis of the defects, and determine if the criteria have been met.
If the criteria have not been met, there are several alternatives:
Gather additional information:
- Produce different reports, such as different defect density reports
- Investigate the process to determine if unexpected conditions contributed to the deviation from the stated test criteria and re-assess the criteria based upon this new knowledge
Suggest scheduling of additional testing:
- Implement new tests to add depth to the test cases
- Implement new test to add greater test coverage
Modify the test criteria:
- Review and assess the risk of changing the criteria POST testing
- Identify a subset of the software that satisfies the test criteria and decide whether it can be deployed.
See "Key Measures of Testing" in Introduction to Test.
Generate the Test Evaluation
Summary
Purpose
|
The last step in the Evaluate Test activity is to generate the test evaluation summary. This is accomplished by assembling the above information into a single report and distributing it to the appropriate workers for review.
