Test Matrix
From dis-Emi-A
A Test Matrix is a means of cataloging Equivalence Classes. It employs a very typical table layout and is useful in situations where a tree form would lead to too many branches, or would poorly sort the branches.
Contents |
Basic Setup
To describe the basic use of a matrix we will use a very simple example, given below.
An image needs to be loaded into the program, and the program supports several image types of various color depths. In each case the image is loaded, so the effect is the same, and as there are many combinations.
Format \ Color 1-Bit 8-Bit 24-Bit RGB 32-Bit RGBA GIF X X JPEG X X X PNG
These format works very well when we have two fixed inputs which can be chosen in combination. As seen above, it even works if not all the combinations are valid or possible, we simply mark them with an X. (For more than two inputs, or open inputs, see below)
Prioritization
The purpose of this matrix is to get a good coverage of the possible inputs without duplicating effort; in this visual form the selection of priorities is relatively easy.
The minimum level of priority is to cover one item in every row, and one item in every column. If you have no X's in your matrix then this would simply be testing each condition along the diagonal. If you have X's however, you may need to produce a snake-like path through the table, such as below.
For the minimum testing we need to cover one item in each column and row, and minimize the number of tests. In our example we can do that as below. Some of the choices are forced in our case (like 24-bit JPEG, or 32-bit PNG), but the other ones, like 1-bit GIF and 8-bit PNG are open. Also notice that we have two items selected in PNG. Such double selections will happen anytime the number of rows and columns are not equal.
Format \ Color 1-Bit 8-Bit 24-Bit RGB 32-Bit RGBA GIF [ ] X X JPEG X X [ ] X PNG [ ] [ ]
Invalid Tests
From the setup it should be clear that we are focusing on testing valid input cases. Including invalid tests becomes troublesome since if the particular test causes an error in the program, then the other tests in the cell have basically not been performed.
As an exception to this, if your X's mark logically invalid input, but are actually possible inputs, then they can be quickly tested to ensure the program produces an error.
Additional Fixed Inputs
A third input would be best handled with a 3-dimensional matrix. That is of course rather hard to do, so an alternate solution is needed.
For a fixed input with a minimum of options, you can insert the options into the table at various locations. These locations should follow the pattern from above (that is, preferably not introduce new test combinations). They also have to make sense, since the option may not make sense at all of the locations.
To continue with the example above we will add a new input called "Preserve Transparency", which has two options "True" or "False". To cover both cases we can extend our matrix as follows:
Format \ Color 1-Bit 8-Bit 24-Bit RGB 32-Bit RGBA GIF [Preserve: True] X X JPEG X X [ ] X PNG [ ] [Preserve=False]
Typically the additional variable should have fewer options than either of the main axes. If it has more, then it is likely a better candidate for the main axes than what you currently have -- this is primarily for formatting purposes.
Four or more inputs
The fourth input can be handled in the same way: put each of the options into one of the already selected boxes in the matrix. Preferably in a cell where there are not so many other inputs already being labeled. This will help keep the matrix easy to read, and starts to broaden the test criteria.
Once you have four or more inputs it starts to make sense to break away from the original one in each row/column criteria. This feature obviously has many combinations and should now be subjected to more than just the minimal testing.
Open Inputs
Inputs which are not fixed at first glance do not play well with the matrix system: the matrix requires fixed inputs. We need someway to translate the open input into a fixed input.
And we have that. The tree form in the Equivalence Classes does exactly that. The tree form takes one input and branches out into relative input classes, eventually leaving us with a fixed number of leaf tests. We then take each valid leaf and place it in the matrix as we would any fixed input.
Note that you use the valid cases. Invalid cases are better handled as independent tests since they tend to cause an error in the program being tested. If the program has an error, then the other items in the same cell can't really said to have been tested, therefore invalid cases are best tested outside of the matrix.
