Guidelines: Test Case

Test Case

A Test case is a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

Topics

• Explanation
• Deriving Test Cases from Use Cases
• Deriving Test Cases from Supplementary Specifications

  • Deriving Test Cases for Performance Tests
  • Deriving Test Cases for Security / Access Tests
  • Deriving Test Cases for Configuration Tests
  • Deriving Test Cases for Installation Tests
  • Deriving Test Cases for other Non-Functional Tests

• Deriving test cases for Unit Tests

• White-box tests
• Black-box tests

• Deriving Test Cases for Product Acceptance Test
• Build Test Cases for Regression Test

Explanation

Nothing has a greater effect on the end-user's satisfaction with the software than a clear view of what the end-user expects so that those expectations can be verified and validated. Test cases reflect the requirements that are to be verified. Verifying these requirements, however, may be done differently and by different testers. For example, executing the software to verify its function and performance may be done by a tester using automated test techniques, the shut-down sequence of a computer system may be done by manual test and observation, while market share and sales, (also product requirements), will be done by measuring product and competitive sales.

Since you may not be able to (or be responsible to) verify all requirements, it is critical for the success of your project to select the most appropriate or critical ones requirements for test. The requirements you choose to verify will be a balance between the cost, risk, and necessity of having the requirement verified.

Identifying the test cases is important for several reasons.

• Test cases form the foundation on which to design and develop test procedures.
• The "depth" of the testing is proportional to the number of test cases. Greater confidence in the quality of the product and test process is gained when the number of test cases increases, since each test case reflects a different scenario, condition, or flow through the product.
• A principal measure of the completeness of test is requirements-based coverage, based on the of the number test cases identified, implemented, and / or executed. A statement such as "95 percent of our critical test cases have been executed and verified" is more significant than stating "We're 95 percent of the way through our tests."
• The scale of the test effort is proportional to the number of test cases. With a comprehensive breakdown of test cases, the timing of succeeding stages of the test cycle can be more accurately estimated.
• The kinds of test design and development, and the resources needed are largely governed by the test cases.

Test cases are often categorized or classified by the type of test or requirement for test they are associated with, and will vary accordingly. Best practice is to develop at least two test cases for each requirement for test:

• a test case to demonstrate the requirement has been achieved, often referred to as a positive test case,
• another test case, reflecting an unacceptable, abnormal, or unexpected condition or data, to demonstrate that the requirement is only achieved under the desired condition, referred to as a negative test cases.

Deriving Test Cases from Use Cases

Test cases for functional testing are derived from the target-of-test's use cases (see Artifact: Use Case). Test cases should be developed for each use-case scenario. The use-case scenarios are identified by describing the paths through the use case that traverse the basic flow and alternate flows start to finish through the use case.  

In the diagram below, for example, each of the different paths through a use case reflecting the basic and alternate flows, are represented with the arrows. The basic flow, represented by the straight, black-line is the simplest path through the use case. Each alternate flow begins with the basic flow and then, dependent upon a specific condition, the alternate flow is executed. Alternate flows may rejoin the basic flow (alternate flows 1 and 3), may originate from another alternate flow (alternate flow 2), or may terminate the use case without rejoining a flow (alternate flows 2 and 4).

Sample Flows of Events for a use case

Following each possible path through the use case in the above diagram, the different use-case scenarios can be identified. Beginning with the basic flow and then combining the basic flow with alternate flows, the following use-case scenarios can be identified: 

NOTE:  For simplicity, Scenarios 5, 6, and 8 only depict a single execution of the loop indicated by Alternate flow 3.    

Deriving the test cases for each scenario is done by identifying the specific condition that will cause that specific use-case scenario to be executed.

For example, suppose the use case depicted in the diagram above stated the following for Alternate Flow 3:

"This flow of events occurs if the dollar amount entered in Step 2 above, "Enter Withdraw Amount" is greater than the current account balance. The system displays a warning message and then rejoins the basic flow at Step 2 "Enter Withdraw Amount" above, where the bank customer can enter a new withdrawal amount."

With this information, you can begin to identify the test cases needed to execute the alternate flow 3:

NOTE: the test cases shown above are very simplistic since no other information was provided. Test cases are rarely this simple. 

A more realistic example of deriving test cases from use cases is provided below.

Example:

Actors and use cases in an ATM machine.

The following table contains the basic flow and some alternate flows for Cash Withdrawal use case in the diagram above:

The following scenarios can be derived from this use case

NOTE:  For simplicity the loops in Alternate flows 3 and 6 (Scenarios 3 and 7), and combinations of loops have not been included in the table above.   

For each of these seven scenarios, test cases need to be identified. Test cases can be identified and managed using matrices or decision tables. A common format is shown below, where each row represent an individual test case, and the columns identify test case information.  In this example, for each test case, there is a test case ID, Condition (or description), and all the data elements participating in the test case (as input or already in the database), and expected result. 

To begin developing the matrix, start by identifying what data elements are required to execute the use-case scenarios. Then, for each scenario, identify at least test case that contains the appropriate condition to execute the scenario. For example, in the matrix below, V (valid) is used to indicate this condition must be VALID for the basic flow to execute and I (invalid) is used to indicate the condition which will invoke the desired alternate flow. "n/a" is used in the table below to indicate that this condition is applicable to the test case.

In the matrix above, the six test cases execute the four scenarios. For the Basic Flow, test case CW1 above is known as a positive test case. It executes the Basic Flow path through the use case without any deviations. Comprehensive testing of the Basic Flow must include negative test cases to ensure that the Basic Flow is taken only when the conditions are correct. These negative test cases are represented by test cases CW2 - 6 (the shaded cell indicates the condition needed to execute the alternate flows). While CW2 - 6 are negative test cases for the Basic Flow, they are positive test cases for Alternate flows 2 - 4, and there is at least one negative test case each of these Alternate Flows (CW1 - the Basic Flow).  

Scenario 4 is an example where having just one positive and one negative test case per scenario is not sufficient. To thoroughly test Scenario 4 - Incorrect PIN, at least three positive test cases (to invoke Scenario 4) are needed:

• the incorrect PIN is entered and there are tries left and this Alternate Flow rejoins the Basic Flow Step 3 - Enter PIN)
• the incorrect PIN is entered and there are no remaining tries left and this Alternate Flow then retains the card and terminates the use case.
• the CORRECT PIN is entered when there are no remaining tries left.  This Alternate Flow rejoins the Basic Flow at Step 5 - Enter Amount. 

Notice, that in the above matrix, no actual values were entered for the conditions (data). An advantage of creating the test case matrix in this manner is that it is easy to see what conditions are being tested. It is also very easy to determine if sufficient test cases have been identified, since you only need to look at Vs and Is (or as done here - shaded cells). Looking at the above table, there are several conditions for which there is no shaded cell, therefore, we are missing test cases, such as for Scenario 6 - No Account or Incorrect Account Type and Scenario 7 - Insufficient Account Balance.

One all the test cases have been identified, they should be reviewed and validated to ensure accuracy, appropriateness, and eliminate redundant or equivalent test cases. See Checkpoints: Test Case for more details.

Upon approval of the test cases, the actual data values can be identified (in the test case implementation matrix) and the test data built (see Guidelines: Test Data for more information regarding test data).

The test cases above are only a few of the test cases needed to verify the Cash Withdraw Use Case for this iteration. Other test cases needed include:

• Scenario 6 - No Account or Incorrect Account Type: Account not found or available
• Scenario 6 - No Account or Incorrect Account Type: Account does not allow withdraws
• Scenario 7 - Insufficient Account Balance: Amount requested greater than amount in account.

In future iterations, when other flows are implemented, test cases will be needed for:

• Invalid cards (card is reported lost, stolen, is not from an accepted bank, has a damaged stripe, etc.)
• Inability to read a card (card reader is jammed, off-line, or malfunctioning)
• Account is closed, frozen, or otherwise unavailable
• Amount in ATM is insufficient or incapable of creating requested amount (different than CW3, in that one denomination is out, but not all)
• Incapable of contacting banking system for approval
• Bank network goes off line, or power failure mid-transaction

When identifying functional test cases, ensure the following:

• sufficient test cases, positive and negative, have been identified for each use-case scenario 
• test cases address any business rules implemented by the use cases, ensuring that there are test cases inside, outside, and at the boundary condition / value for the business rule
• test cases address any sequencing of events or actions, such as those identified in the sequence diagrams in the design model, or user interface object states or conditions.
• test cases address any special requirements defined for the use case, such minimum/maximum performance, sometimes combined with minimum/maximum loads or data volumes during the execution of the use cases.

See also Guidelines: Test Data for additional information regarding the test data.

Deriving Test Cases from Supplementary Specifications

Not all requirements for a target-of-test will be reflected in use cases. Non-functional requirements, such as performance, security and access, and configuration requirements specify additional behaviors or characteristics of the target-of-test. The Supplementary Specification is the primary source for deriving test cases for these additional behaviors.

Below are described guidelines for deriving these additional test cases:

• Deriving Test Cases for Performance Tests
• Deriving Test Cases for Security / Access Tests
• Deriving Test Cases for Configuration Tests
• Deriving Test Cases for Installation Tests
• Deriving Test Cases for other non-functional tests

Deriving Test Cases for Performance Test

The primary input for performance test cases are the Supplementary Specifications which contain the non-functional requirements (see Artifact: Supplementary Specifications). Use the following guidelines when deriving test cases for performance test:

• ensure there is at least one test case identified for each statement in the supplementary specification which states a performance criteria. Performance criteria are usually expressed as time per transaction, number of transactions / users, or percentiles.
• ensure there is at least one test case identified for each critical use case. Critical use cases are those identified in the above statements and / or in the workload analysis document (see Artifact: Workload Analysis Document) that must be evaluated using performance measures.

As with test cases for functional tests, there will typically be more than one test case per use case / requirement. It is more common that there will be one test case that is below the performance threshold value another at the threshold value, and a third test case above the threshold value.

In addition to the above performance criteria, ensure that you identify the specify conditions that affect response times, including:

• Size of the database - how many records exist?
• Workload - number and type of simultaneous end-users, number and type of simultaneous transactions being performed
• Environment characteristics (hardware, netware, software configuration)

Capture test cases for performance test in matrices similar to those used for function test.

See also Guidelines: Test Data for additional information regarding the test data.

Below are some examples for the different types of Performance Tests:

For Load Test:

For Stress Test:

Deriving Test Cases for Security / Access Tests

Actors and use cases describe the interaction between external users of the system and the actions performed by the system to yield a value to a particular actor. Complex systems contain many actors and it is critical that we develop test cases to ensure that only those actors specified to execute the use cases can do so. This is especially true if there are differences in the use case flow of events based upon actor type.

For example, in the ATM use cases, different use case flow of events may be executed for the actor "Bank Customer" if their card and account is from the bank that owns the ATM versus the "Bank Customer" who uses a bank card (and account) from a competing bank, or tries to use a non-participating bank card.

Follow the same guidelines as listed above for functional test cases.

See also Guidelines: Test Data for additional information regarding the test data.

Example test cases for Security and Access:

Deriving Test Cases for Configuration Tests

In typical distributed systems there can be many allowed combinations of hardware and software that will be supported. Testing should be performed to verify that the target-of-test functions or performs acceptably in different configurations, such as with different operating systems, browsers, or CPU speeds. Furthermore, testing also needs to cover combinations of components to uncover defects that come from interactions of the different components, for example, ensuring that the version of DLLs installed by one application do not conflict the versions of the same DLLs expected by another application.

To derive test cases for configuration testing, use the following guidelines:

• Ensure there is at least one test case identifying each critical configuration. This is done by identifying the required hardware and software configurations for the target-of-test's environment and prioritizing the configurations, ensuring the most common ones are tested first, including:
  • Printer support
  • Network connections - local and wide area networks
  • Server configurations - server drivers, server hardware
  • Other software installed on the desktop and / or servers
  • Software versions for all installed software
• Ensure there is at least one test case for each configuration likely to have problems. These may include:
  • Hardware with the lowest performance.
  • Co-resident software that has a history of compatibility problems.
  • Clients accessing the server over slowest possible LAN/WAN connection.
  • Insufficient resources (slow CPU speed, minimum memory or resolution, disk space, etc.)

Deriving Test Cases for Installation Tests

Installation testing needs to verify that the target-of-test can be installed under all possible installation scenarios. Installation scenarios may include installing the target-of-test for the first time, or installing a newer version or build of the target-of-test (onto a machine containing the older version). Install testing should also ensure that the target-of-test performs acceptably when abnormal conditions are encountered, such as insufficient disk space.

The test cases should cover installation scenarios for the software including:

• Distribution media, for example, diskettes, CD-ROM, or file server.
• New installation.
• Complete installation.
• Custom installations.
• Upgrade installations.

Installation programs for client-server software have a specialized set of test cases. Unlike host-based systems, the installation program is typically divided between the server and the client. Therefore, it is important that installation testing perform the installation of all components of the target-of-test, including the client, middle tiers, and servers.

Deriving Test Cases for other Non-Functional Tests

Ideally, you should find all the necessary input to derive test cases in the Use-Case Model, the Design Model, and the Supplementary Specification artifacts. It is, however, not uncommon that you at this point need to complement what is found there.

Examples would be:

• Test cases for Operational Tests (to verify that the software works when in use for a "long time" between failures).
• Test cases that investigate performance bottlenecks, volume capabilities of the system, or stress the target-of-test to failure.

In most cases, you can find these test cases by creating variants or aggregates of the test cases you've derived from those previously identified.

Deriving Test Cases for Unit Test

Unit testing requires testing both the unit's internal structure and its behavioral characteristics. Testing the internal structure requires a  knowledge of how the unit is implemented, and tests based upon this knowledge are known as white-box tests. Testing a unit's behavioral characteristics focuses on the external observable behaviors of the unit without knowledge or regard its implementation. Tests based upon this approach are referred to as black-box tests. Deriving test cases based upon both approaches are described below.

White-Box Tests

Theoretically, you should test every possible path through the code. Achieving such a goal, in all but very simple units, is either impractical or almost impossible. At the very least you should exercise every decision-to-decision path (DD-path) at least once, resulting in executing all statements at least once. A decision is typically an if-statement, and a DD-path is a path between two decisions.

To get this level of test coverage, it is recommended that you choose test data so that every decision is evaluated in every possible way. Toward that end, the test cases should make sure that:

• Every Boolean expression is evaluated to true and false. For example the expression (a<3) OR (b>4) evaluates to four combinations of true/false
• Every infinite loop is exercised at least zero times, once, and more than once.

Use code-coverage tools to identify the code not exercised by your white box testing. Reliability testing should be done simultaneously with your white-box testing.

Example:

Assume that you perform a structure test on a function member in the class Set of Integers. The test - with the help of a binary search - checks whether the set contains a given integer.

The member function and its corresponding flowchart. Dotted arrows illustrate how you can use two test cases to execute all the statements at least once.

Theoretically, for an operation to be thoroughly tested, the test case should traverse all the combinations of routes in the code. In member, there are three alternative routes inside the while-loop. The test case can traverse the loop either several times or not at all. If the test case does not traverse the loop at all, you will find only one route through the code. If it traverses the loop once, you will find three routes. If it traverses twice, you will find six routes, and so forth. Thus, the total number of routes will be 1+3+6+12+24+48+à, which in practice, is an unmanageable number of route combinations. That is why you must choose a subset of all these routes. In this example, you can use two test cases to execute all the statements. In one test case, you might choose Set of Integers = {1,5,7,8,11} and t = 3 as test data. In the other test case, you might choose Set of Integers = {1,5,7,8,11} and t = 8.

See Guidelines: Unit Test for additional information

Black-Box Tests

The purpose of a black-box test is to verify the unit's specified behavior without looking at how the unit implements that behavior. Black-box tests focus and rely upon the unit's input and output.

Equivalence partitioning is a technique for reducing the required number of tests. For every operation, you should identify the equivalence classes of the arguments and the object states. An equivalence class is a set of values for which an object is supposed to behave similarly. For example, a Set has three equivalence classes: empty, some element, and full.

Use code-coverage tools to identify the code not exercised by your white box testing. Reliability testing should be done simultaneously with your black-box testing.

The next two subsections describe how to identify test cases by selecting test data for specific arguments.

Test Cases based upon Input Arguments

An input argument is an argument used by an operation. You should create test cases by using input arguments for each operation, for each of the following input conditions: 

• Normal values from each equivalence class.
• Values on the boundary of each equivalence class.
• Values outside the equivalence classes.
• Illegal values.

Remember to treat the object state as an input argument. If, for example, you test an operation add on an object Set, you must test add with values from all of Set's equivalence classes, that is, with a full Set, with some element in Set, and with an empty Set.

Test Cases based upon Output Arguments

An output argument is an argument that an operation changes. An argument can be both an input and an output argument. Select input so that you get output according to each of the following.

• Normal values from each equivalence class.
• Values on the boundary for each equivalence class.
• Values outside the equivalence classes.
• Illegal values.

Remember to treat the object state as an output argument. If for example, you test an operation remove on a List, you must choose input values so that List is full, has some element, and is empty after the operation is performed (test with values from all its equivalence classes).

If the object is state-controlled (reacts differently depending on the object's state), you should use a state matrix such as the one in the following figure.

A state matrix for testing. You can test all combinations of state and stimuli on the basis of this matrix.

See Guidelines: Unit Test for additional information

Deriving Test Cases for Product Acceptance Test

Product acceptance testing is the final test action prior to deploying the software. The goal of acceptance testing is to verify that the software is ready and can be used by the end-users to perform those functions and tasks the software was built to do. Product acceptance testing often involves more than execution of the software for readiness, it also involves all product artifacts delivered to the customer(s), such as training, documentation, and packaging. 

Deriving test cases for the software artifact(s) is done in the manner described in the sections above. Depending upon the degree and of formality of the product acceptance test, the test cases will either be the same or similar to those identified above (formal), or a sub-set (informal). Independent of the depth of the test cases, agreement on the test cases and Product Acceptance Plan should be reached at before product testing is implemented and executed.

Evaluating the non-software artifact(s) varies greatly dependent upon the artifact being evaluated. Refer to each specific non-software artifactÆs Guidelines and Checklists for information regarding what / how to evaluate it.

Build Test Cases for Regression Test

Regression testing compares two builds or versions of the same target-of-test and identifies differences as potential defects. It thus assumes that a new version should behave like an earlier one and ensures that defects have not been introduced as a result of the changes.

Ideally, you would like all the test cases in one iteration to be used as test cases in the later iterations. The following guidelines should be used to identify, design, and implement test cases that maximize the value of regression testing and re-use, while minimizing maintenance:

• Ensure the test case identify only the critical data elements (those needed to create / support the condition being tested
• Ensure each test case describes or represents a unique set of inputs or sequence of events that results in a unique behavior by the target-of-test
• Eliminate redundant or equivalent test cases
• Group together test cases which have the same target-of-test initial state and state of the test data

Rational Unified Process