Guidelines: Test Plan


Test Plan
The test plan contains information about the purpose and goals of testing within the project. Additionally, the test plan identifies the strategies to be used to implement and execute testing, and the resources needed.

The purpose of the test plan is to communicate the intent of the testing activities. It is critical that this document be created as early as possible. Generating this artifact early in one of the first iterations of the Elaboration phase would not be too early. It may be desirable to develop the test plan iteratively, adding sections as the information is available.

Care should be taken in clearly communicating the scope of testing, the requirements for tests, the test strategies, and the resource needs. This information identifies the purpose and boundaries of the test effort, what will be tested, how it will be tested, and what resources are needed for testing. Stating this information clearly, will expedite the review, feedback, and approval of the test effort.

At the outset of the project, a test plan identifying the overall intended testing for the project should be created, called the “Master Test Plan.” As each iteration is planned, a more precise “Iteration Test Plan” is created (or several test plans, organized by type of test), containing only the data (requirements for tests, test strategies, resources, etc.) that pertain to the iteration. Alternately, this information may be included in the Iteration Plan, if it does not make the iteration plan too difficult to manage or use. 

Below are some guidelines to better identify and communicate the requirements for test, test risks and priorities, and test strategies.

Identifying Requirements for Tests

Requirements for tests identify what will be tested. They are the specific target of a test. There are a few general rules to apply when deriving requirements for tests:

  • The requirement for test must be an observable, measurable behavior. If the requirement for test cannot be observed or measured, it can’t can be assessed to determine if the requirement has been satisfied.
  • There is not a one-to-one relationship between each use case or supplemental requirement of a system and a requirement for test. Use cases will often have more than one requirement for test, while some supplemental requirements will derive one or more requirements for tests and others will derive none (such as marketing or packaging requirements).

The requirements for tests may be derived from many sources, including use cases, use-case models, supplemental specifications, design requirements, business cases, interviews with end-users, and the software architecture document. All of these should be reviewed to gather information that is used to identify the requirements for tests.

Requirements for Functional Tests

Functional requirements for tests, as their name implies, are derived from descriptions of the target-of-test’s functional behaviors. At a minimum, each use case should derive at least one requirement for test. A more detailed list of requirements for tests would include at least one requirement for test for each use case flow of events.

Requirements for Performance Tests

Performance requirements for tests are derived from the target-of-test’s specified performance behaviors. Typically, performance is stated as a measure of response time and/or resource usage, as measured under various conditions, including

  • different workloads and/or system conditions
  • different use cases
  • different configurations

Requirements for performance are described in the Supplementary Specifications. Review these materials, paying especial attention to statements that include the following:

  • statements of time, such as response time or timing profiles
  • statements indicating that a number of events or use cases must occur in a stated period of time
  • statements comparing the behavior of one item to another
  • statements comparing the application behavior on one configuration to that of another
  • operational reliability (mean time to failure or MTTF) over a period of time
  • configurations or constraints

You should derive at least one requirement for test for each statement in the specification which reflects information such that listed above.

Requirements for Reliability Tests

Reliability requirements for tests are derived from several sources, typically described in Supplementary Specifications, User-Interface Guidelines, Design Guidelines, and Programming Guidelines.

Review these artifacts and pay especial attention to statements that include the following:

  • statements reliability or resistance to failure, run-time errors (such as memory leaks)
  • statements indicating code integrity and structure (compliance to language and syntax)
  • statements regarding resource usage

At least one requirement for test should be derived from each statement in the artifacts that reflects information listed above.

Successful testing requires that the test effort successfully balance factors such as resource constraints and risks. To accomplish this, the test effort should be prioritized so that the most important, significant, or riskiest use cases or components are tested first. To prioritize the test effort, a risk assessment and operational profile are performed and used as the basis for establishing the test priority.

The following sections describe how to determine test priority.

Assess Risk and Establish Test Priorities

Identifying the requirements for tests is only part of identifying what will be tested. Prioritizing what will be tested and in what order should also be performed. This step is done for several reasons, including:

  • ensure the test efforts are focused on the most appropriate requirements for test
  • ensure the most critical, significant, or riskiest requirements for tests are addressed as early as possible
  • ensure that any dependencies (sequence, data, etc.) are accounted for in the testing

There are three steps to assessing risk and establishing the test priorities:

  • Assess Risk
  • Determine Operational Profile
  • Establish Test Priority

Guidelines for each of these three steps are provided below:

Assess Risk

Establishing the priority for test begins with the assessment of risk. Use cases or components that pose the greatest risk due to failure or have a high probability of failure should be amongst the first use cases tested.

Begin by identifying and describing the risk magnitude indicators that will be used, such as:

  • H – high risk, not tolerable. Severe external exposure. The company will suffer great financial losses, liability, or un-recoverable loss of reputation.
  • M – medium risk, tolerable, but not desirable. Minimal external exposure, the company may suffer financially, but there is limited liability or loss of reputation.
  • L – low risk, tolerable. Little or no external exposure, company has little or no financial loss or liability. Company’s reputation unaffected.

After identifying the risk magnitude indicators, list each use case or component in the target-of-test. For each use case or component in your list, identify a risk magnitude indicator, and justify (in a brief statement) the value you selected.

There are three perspectives that can be used for assessing risk:

  • Effect – the impact or consequence of a specified use case (requirement, etc.) failing
  • Cause – an undesirable outcome caused by the failure of a use case
  • Likelihood – the probability of a use case failing.

Select one perspective, identify a risk magnitude indicator and justify your selection. It is not necessary to identify an indicator for each risk perspective. However, it suggested that, if a low indicator was identified, try evaluating the item from a different risk perspective to ensure the item is really a low risk.

Below are greater details on assessing risk by these three perspectives.

Effect

To assess risk by Effect, identify a condition, event, or action and try to determine its impact. Ask the question:

“What would happen if ___________?”

For example:

  • “What would happen if while installing the new software, the system runs out of disk space?”
  • “What would happen if the Internet connection is lost during an inquiry transaction?”
  • “What would happen if the Internet connection was lost during a purchase transaction?”
  • “What would happen if the user enters an unexpected value?”

Below is a sample justification matrix for these items:

Description Risk Mitigation Factor Justification
Insufficient disk space during install H Installing the software provides the user with the first impression of the product. Any undesirable outcomes, such as those listed below would degrade the user’s system, the installed software, and communicate a negative impression to the user:

  • software is partially installed (some files, some registry entries), which leaves the installed software in an unstable condition, or
  • the installation halts leaving the system in an unstable state
Internet connection lost during inquiry L No damage resulting from the lost connection is done to the data or database. It is recognized that a lost connection may communicate a negative impression to the user.
Internet connection lost during purchase H Any lost connections or transactions that result in the outcomes listed below are unacceptable, as they increase the overhead costs and decrease profits:

  • corrupted database
  • partial order
  • lost data or order
  • multiple orders (replicated)
Unexpected value entered H Any transactions that result in the outcomes listed below are unacceptable:

  • corrupted database
  • inaccurate data

Assessing risk by Cause is the opposite of by Effect. Begin by stating an undesirable event or condition, and identify the set of events that could have permitted the condition to exist. Ask a question such as:

“How could ___________ happen?

For example:

  • “How could only some of the files be on the system and not all the registry entries made?”
  • “How could a transaction not be reflected properly in the central database?
  • “How could the billing cycle statement reflect only some of the records in the database that fulfill the desired criteria?”

Below is a sample justification matrix for these items:

Description Risk Mitigation Factor Justification
Missing / application files and registry entries H Renders the application (and potentially the system) un-usable. Installation is the first view of the application seen by the users. If installation fails, for any reason, the user views the software unfavorably.

Possible causes of this condition include:

  • the installation process did not install all the files and update the registry correctly
  • the installation process halted due to user intervention (cancel or exit)
  • the installation process halted due to software / hardware intervention (insufficient disk space, unsupported configuration, etc.)
  • the installation process halted due to unknown conditions
  • the user deleted files / registry entries

Of these causes, only the last one cannot be detected and handled by the installation process.

Partial order H Partial orders cannot be fulfilled, resulting in lost revenue and lost customers.

Possible causes include:

  • Internet connection lost due to user action (disconnect modem, turn off PC, etc.)
  • Internet connection lost due to IP
  • Internet connect lost due to employee action (disconnect modem, turn off power to servers, etc.)
Corrupt data / database H Corrupt data cannot be tolerated for any reason.

Possible causes include:

  • Transaction that writes to the database not completed / committed due to user intervention
  • Transaction that writes to the database not completed / committed due to lost Internet connection
  • User enters invalid data in transaction
  • Database access methods / utilities
  • Database not properly populated (when initially instantiated)
Replicated orders H Replicated orders increase the company overhead and diminish profits via the costs associated with shipping, handling, and restocking.

Possible causes include:

  • Transaction that writes order to the database replicated due to user intervention, user enters order twice – no confirmation of entry
  • Transaction that writes order to the database replicated due to non-user intervention (recovery process from lost Internet connection, restore of database)
Inaccurate data for an order H Any orders that cannot be completed or incur additional overhead costs are not acceptable.

Possible causes include:

  • Order transaction is not completed / committed due to user intervention
  • Order transaction is not completed / committed due to lost Internet connection
  • User enters invalid data
Wrong number of records reflected in statement H Business decisions and accounts receivable are dependent upon the accuracy of these reports.

Possible causes include:

  • Incorrect search / select criteria
  • Incorrect SQL statement
  • Corrupt data in database
  • Incorrect data in database

Assessing risk by Likelihood is to determine the probability that a use case (or component implementing a use case) will fail. The probability is usually based on an external factors such as:

  • Failure rate(s) and / or density
  • Rate of change
  • Complexity
  • Origination / Originator

It should be noted, that when using this risk perspective, the risk magnitude indicators are related to the probability of a failure, not the effect or impact the failure has on the organization as was used in assessing risk by Effect and Cause.

Correlations between these factors and the probability of a failure exist, as identified below:

External Factor Probability
Failure discovery rate
and / or density
The probability of a failure increases as the failure discovery rates or density increases. Defects tend to congregate, therefore, as the rate of discovery or the number of defects (density) increases in a use case or component, the probability of finding another defect also increases. Discovery rates and density from previous releases should also be considered when assessing risk using this factor, as previous high discovery rates or densities indicate a high probability of additional failures.
Rate of change The probability of a failure increases as the rate of change to the use case or component increases. Therefore, as the number of changes increases, so too does the probability that a defect has been introduced. Every time a change is made to the code, there is the risk of “injecting” another defect in it.
Complexity The probability of a failure increases as the measure of complexity of the use case or component increases.
Origination / Originator Knowledge and experience of where the code originated and by whom can increase or decrease the probability of a failure. The use of third party components typically decreases the probability of failure. However, this is only true if the third party component has been certified (meets your requirements, either through formal test or experience).

The probability of failure typically decreases with the increased knowledge and skills of the implementer. However, such factors as the use of new tools, technologies, or acting in multiple roles may increase the probability of a failure even by the best team members.

For example:

  • Installing the new software
  • “Historically we’ve found many defects in the components used to implement use cases 1, 10, and 12, and our customers requested many changes in use case 14 and 19.”

Below is a sample justification matrix for these items:

Description Risk Mitigation Factor Justification
Installing new software H We are writing our own installation utility.

Renders the use of the application un-usable. Installation is the first view of the application seen by the users. If installation fails, for any reason, the user views the software unfavorably.

Installing new software L We are using a commercially successful installation utility.

While failed installation renders the use of the application un-usable, the installation utility selected is from a vendor that has achieved the number one market share with their product and has been in business for over four years. Our evaluation of their indicates that the product meets our needs and clients are satisfied with their product, the vendor, and their level of service and support.

High failure discovery rates / defect densities in use cases 1, 10, 12. H Due to the previous high failure discovery rates and defect density use cases 1, 10, and 12 are considered high risk.
Change Requests in use cases 14 and 19. H A high number of changes to these use cases increases the probability of injecting defects into the code.

The next step in assessing risk and establishing a test priority is to determine the target-of-test’s operational profile.

Begin by identifying and describing the operational profile magnitude indicators that will be used, such as:

  • H – used quite frequently, many times per period or by many actors or use cases.
  • M – used frequently, several times per period or by several actors or use cases.
  • L – infrequently used or used by very few actors or use cases.

The operational profile indicator you select should be based upon the frequency a use case or component is executed, including:

  • the number of times ONE actor (or use case) executes the use case (or component) in a given period of time, or
  • the number of ACTORS (or use cases) that execute the use case (or component)

Typically, the greater the number of times a use case or component is used, the higher the operational profile indicator.

After identifying the operational profile magnitude indicators to be used, list each use case or component in the target-of-test. Determine an operational profile indicator for each item in your list and a state your justification for the indicator value.

Examples:

  • Installing new software
  • Ordering items from the on-line catalog
  • Customers inquiring about their order on-line after order is placed
  • Item selection dialog

Description Operational Profile Factor Justification
Installing new software H Performed once (typically), but by many users. Without installation however, application is unusable.
Ordering items from the catalog H This is the most common use case executed by users.
Customers inquiring about orders L Few customers inquire about their orders after they are placed
Item selection dialog H This dialog is used by customers for placing orders and by inventory clerks to replenish stock.

Establish Test Priority

The last step in the assessing risk and establishing a test priority is to establish the test priority.

Begin by identifying and describing the test priority magnitude indicators that will be used, such as:

  • H – must be tested
  • M – should be tested, will test only after all H items are tested
  • L – might be tested, but not until all H and M items have been tested

After identifying the test priority magnitude indicators to be used, list each use case or component in the target-of-test. Determine a test priority indicator for each item in your list and a state your justification. Below are some guidelines for determining a test priority indicator.

Consider the following when determining the test priority indicators for each item:

  • the risk magnitude indicator value you identified earlier
  • the operational profile magnitude value you identified earlier
  • the actor descriptions (are the actors experienced?, tolerant of work-arounds?, etc.)
  • contractual obligations (will the target-of-test be acceptable if a use case or component is not delivered?)

Strategies for establishing a test priority include:

  • Use the highest assessed factor (risk, operational profile, etc.) value for each item as the overall priority.
  • Identify one assessed factor (risk, operational profile, other) as being the most significant and use that factor’s value as the priority.
  • Use a combination of assessed factors to identify the priority.
  • Using a weighting schema where individual factors are weighed, and their values and priority calculated based upon the weight.

Examples:

  • Installing new software
  • Ordering items from the on-line catalog
  • Customers inquiring about their order on-line after order is placed
  • Item Selection Dialog

Priority when the highest assessed value is used to determine priority:

Item Risk Operational Profile Actor Contract Priority
Installing new software H H L H H
Ordering items from catalog H H H H H
Customer Inquiries L L L L L
Item Selection Dialog L H L L H


Item Risk Operational Profile Actor Contract Priority
Installing new software H H L H H
Ordering items from catalog H H H H H
Customer Inquiries L L L L L
Item Selection Dialog L H L L L

(Note: in the matrix below, H = 5, M = 3, and L = 1. A Total Weighted value greater than 30 is a High priority test item, values between 20 and 30 inclusive are a Medium priority, and values less than 20 are Low).

Item Risk (x 3) Operational Profile (x 2) Actor (x 1) Contract (x 3) Weighted Value Priority
Installing new software 5 (15) 5 (10) 1 (1) 5 (15) 41 H (2)
Ordering items from catalog 5 (15) 5 (10) 5 (5) 5 (15) 45 H (1)
Customer Inquiries 1 (3) 1 (2) 1 (1) 1 (3) 9 L (4)
Item Selection Dialog 1 (3) 5 (10) 1 (1) 1 (3) 17 L (3)

Test Strategy

The Test Strategy describes the general approach and objectives of a specific test effort.

A good test strategy should contain the following:

Type of Test and Objective

State clearly the type of test being implemented and the objective of the test. Explicitly stating this information reduces confusion and minimizes misunderstandings (especially since some tests may look very similar). The objective should state clearly why the test is being executed.

Examples:

“Functional Test. The functional test focuses on executing the following use cases implemented in the target-of-test, from the user interface.”

“Performance Test. The performance test for the system will focus on measuring response time for use cases 2, 4, and 8 – 10. For these tests, a workload of one actor, executing these use cases without any other workload on the test system will be used.”

“Configuration Test. Configuration testing will be implemented to identify and evaluate the behavior of the target-of-test on three different configurations, comparing the performance characteristics to our benchmark configuration.”

Test Stage

Clearly state the stage in which the test will be executed. Identified below are the stages in which common test are executed:

 
Stage of Test
Type of Tests Unit Integration System Acceptance
Functional Tests

(Configuration, Function, Installation, Security, Volume)

X X X X
Performance Tests

(performance profiles of individual components)

X X (X)

optional or when system performance tests disclose defects

 
Performance Tests

(Load, Stress, Contention)

 
 
X X
Reliability

(Integrity, Structure)

X X (X)

optional or when others tests disclose defects

 

Technique

The technique should describe how testing will be implemented and executed. Include what will be tested, the major actions to be taken during test execution, and the method(s) used to evaluate the results.

Example:

Functional Test:

  • For each use case flow of events, a representative set of transactions will be identified, each representing the actions taken by the actor when the use case is executed.
  • A minimum of two test cases will be developed for each transaction; one test case to reflect the positive condition and one to reflect the negative (unacceptable) condition.
  • In the first iteration, use cases 1 – 4, and 12 will be tested, in the following manner:
    • Use Case 1:
      • Use Case 1 begins with the actor already logged into the application and at the main window, and terminates when the user has specified SAVE.
      • Each test case will be implemented and executed using Rational Robot.
      • Verification and assessment of execution for each test case will be done using the following methods:
        • Test script execution (did each test script execute successfully and as desired?)
        • Window Existence, or Object Data verification methods (implemented in the test scripts) will be used to verify that key windows display and specified data is captured / displayed by the target-of-test during test execution.
        • The target-of-test’s database (using Microsoft Access) will be examined before the test and again after the test to verify that the changes executed during the test are accurately reflected in the data.

Performance Test:

  • For each use case, a representative set of transactions, as identified in the workload analysis document will be implemented and executed using Rational Suite PerformanceStudio (vu scripts) and Rational Robot (GUI scripts).
  • At least three workloads will be reflected in the test scripts and test execution schedules including the following:
    • Stressed workload: 750 users (15 % managers, 50 % sales, 35 % marketing)
    • Peak workload: 350 users (10 % managers, 60 % sales, 30 % marketing)
    • Nominal workload: 150 users (2 % managers, 75% sales, 23 % marketing)
  • Test scripts used to execute each transaction will include the appropriate timers to capture response times, such as total transaction time (as defined in the workload analysis document), and key transaction activity or process times.
  • The test scripts will execute the workloads for one hour (unless noted differently by the workload analysis document).
  • Verification and assessment of execution for each test execution (of a workload) will include:
    • Test execution will be monitored using state histograms (to verify that the test and workloads are executing as expected and desired)
    • Test script execution (did each test script execute successfully and as desired?)
    • Capture and evaluation of the identified response times using the following reports:
      • Performance Percentile
      • Response Time

Completion Criteria

Completion criteria are stated to for two purposes:

  • identify acceptable product quality
  • identify when the test effort has been successfully implemented

A clear statement of completion criteria should include the following items:

  • function, behavior, or condition being measured
  • method of measurement
  • criteria or degree of conformance to measurement

Example 1

  • All planned test cases have been executed
  • All identified defects have been addressed to an agreed upon resolution
  • All planned test cases have been re-executed and all known defects have been addressed as agreed upon, and no new defects have been discovered

Example 2

  • All high priority test cases have been executed.
  • All identified defects have been addressed to an agreed upon resolution.
  • All Severity 1 or 2 defects have been resolved (status = fixed or postponed).
  • All high priority test cases have been re-executed and all known defects have addressed as agreed upon, and no new defects have been discovered.

Example 3

  • All planned test cases have been executed.
  • All identified defects have been addressed to an agreed upon resolution.
  • All Severity 1 or 2 defects have been resolved (status = verified or postponed).
  • All high priority test cases have been re-executed and all known defects have addressed as agreed upon, and no new defects have been discovered.

Special Considerations

This section should identify any influences or dependencies which may impact or influence the test effort described in the test strategy. Influences might include:

  • human resources (such as availability or need for non-test resources to support / participate in test)
  • constraints, (such as equipment limitations or availability, or the need / lack of special equipment)
  • special requirements, such as test scheduling or access to systems

Examples:

  • Test databases will require the support of a database designer / administrator to create, update, and refresh test data.
  • System performance testing will use the servers on the existing network (which supports non-test traffic). Testing will need to be scheduled after hours to ensure no non-test traffic on the network.
  • The target-of-test must synchronize the legacy system (or synchronization simulated) for full functional testing to be implemented and executed

Feedback © 2014 Polytechnique Montreal