Skip to content

Page271

Software Testing Levels

It is usually helpful to approach the challenge of testing software from multiple angles, addressing various testing levels, from low to high. The software testing levels of Unit Testing, Installation Testing, Integration Testing, Regression Testing, and Acceptance Testing are designed to accomplish that goal:

  • Unit Testing: Low-level tests of software components, such as functions, procedures, or objects.
  • Installation Testing: Testing software as it is installed and first operated.
  • Integration Testing: Testing multiple software components as they are combined into a working system. Subsets may be tested, or Big Bang integration testing tests all integrated software components.
  • Regression Testing: Testing software after updates, modifications, or patches.
  • Acceptance Testing: Testing to ensure the software meets the customer’s operational requirements. When this testing is done directly by the customer, it is called User Acceptance Testing.

Fuzzing

Fuzzing (also called fuzz testing) is a type of black box testing that submits random, malformed data as inputs into software programs to determine if they will crash. A program that crashes when receiving malformed or unexpected input is likely to suffer from a boundary checking issue and may be vulnerable to a buffer overflow attack.

Fuzzing is typically automated, repeatedly presenting random input strings such as command line switches, environment variables, and program inputs. Any program that crashes or hangs has failed the fuzz test.

Fuzzing can be considered a particular type of dynamic testing. Fuzzers are simply used to automate providing input to the application. Many people commonly associate fuzzers specifically with uncovering simple buffer overflow conditions. However, advanced and custom fuzzers will do more than simply provide a tremendous volume of input to an application. Fuzzers can and have been used to uncover much more complex flaws than the traditional buffer overflow flaws.

Combinatorial Software Testing

Combinatorial software testing is a black box testing method that seeks to identify and test all unique combinations of software inputs. An example of combinatorial software testing is pairwise testing (also called all pairs testing).

NIST gives the following example of pairwise testing (see http://csrc.nist.gov/groups/SNS/acts/documents/kuhn-kacker-leli-hunter09.pdf): “Suppose we want to demonstrate that a new software application works correctly on PCs that use the Windows or Linux operating systems, Intel or AMD processors, and the IPv4 or IPv6 protocols. This is a total of 2 × 2 × 2 = 8 possibilities but, as (Table 7.1) shows, only four tests are required to test every component interacting with every other component at least once. In this most basic combinatorial method, known as pairwise testing, at least one of the four tests covers all possible pairs (t = 2) of values among the three parameters” [8].

Table 7.1 NIST Pairwise Testing Example [8].

Test Case OS CPU Protocol
1 Windows Intel IPv4
2 Windows AMD IPv6
3 Linux Intel IPv6
4 Linux AMD IPv4