Activity: Execute Test

Execute Test Procedures

To execute the tests, the following steps should be followed:

• Set-up the test environment to ensure that all the needed components (hardware, software, tools, data, etc.) have been implemented and are in the test environment.
• Initialize the test environment to ensure all components are in the correct initial state for the start of testing.
• Execute the test procedures.

Note: executing the test procedures will vary dependent upon whether testing is automated or manual.

• Automated testing: The test scripts created during the Implement Test activity are executed.
• Manual execution: The structured test procedures developed during the Design Test activity are used to manually execute test.

Evaluate Execution of Test

The execution of testing ends or terminates in one of two conditions: 

• Normal: all the test procedures (or scripts) execute as intended and to completion. 

If testing terminates normally, then continue with Verify Test Results:

• Abnormal or premature: the test procedures (or scripts) did not execute completely or as intended. When testing ends abnormally, the test results may be unreliable. The cause of the abnormal / premature termination needs to be identified, corrected, and the tests re-executed before any additional test activities are performed.  

If testing terminates abnormally, continue with Recover From Halted Tests.

Verify Test Results

Upon the completion of testing, the test results should be reviewed to ensure that the test results are reliable and reported failures, warnings, or unexpected results were not caused by external influences (to the target-of-test), such as improper set-up or data.  

The most common failures reported when test procedures and test scripts execute completely, and their corrective actions are given below:

• Test verification failures - this occurs when the actual result and the expected result do not match. Verify that the verification method(s) used focus only on the essential items and / or properties and modify if necessary.
• Unexpected GUI windows - this occurs for several reasons. The most common is when a GUI window other than the expected one is active or the number of displayed GUI windows is greater than expected. Ensure that the test environment has been set-up and initialized as intended for proper test execution.
• Missing GUI windows - this failure is noted when a GUI window is expected to be available (but not necessarily active) and is not. Ensure that the test environment has been set-up and initialized as intended for proper test execution. Verify that the actual missing windows are / were removed from the target-of-test.

If the reported failures are due to errors identified in the test artifacts, or due to problems with the test environment, the appropriate corrective action should be taken and the testing re-executed.  For additional information, see "Recover From Halted Tests" below.

If the test results indicate the failures are genuinely due to the target-of-test, then the Execute Test Activity is complete and typically either the Activity: Submit Change Request or Activity: Fix a Defect should now be performed.

Recover From Halted Tests

There are two major types of halted tests:

• Fatal errors - the system fails (network failures, hardware crashes, etc.)
• Test Script Command Failures - specific to automated testing, this is when a test script cannot execute a command (or line of code).

Both types of abnormal termination to testing may exhibit the same symptoms:

• many unexpected actions, windows, or events occur while the test script is executing
• test environment appears unresponsive or in an undesirable state (such as hung or crashed).

To recover from halted tests, do the following:

• determine the actual cause of the problem
• correct the problem
• re-set-up test environment
• re-initialize test environment
• re-execute tests