PICT (Pairwise Independent Combinatorial Testing)

PICT (Pairwise Independent Combinatorial Testing) is a command-line tool used to generate test cases and configurations. Instead of performing exhaustive testing of every possible combination of parameters—which can result in thousands of combinations—PICT generates a compact set of test cases that provide comprehensive combinatorial coverage by testing all possible pairs of values.

To use PICT, you prepare a model file that details the parameters of the interface or data you want to test. Each parameter in the model has a limited number of possible values. PICT then processes this model to output a manageable suite of test cases that cover all pairs of values across all parameters.

Weights allow you to suggest a preference for certain parameter values. Weights are positive integers.

Usage: Value (Weight)

Important Notes:

• Weights are opportunistic hints, not guarantees. PICT's primary goal is to minimize the number of test cases while maintaining coverage. Weights are only used when multiple valid combinations exist that satisfy the coverage criteria.
• A weight does not mean a value will appear $N$ times more often; it means it is $N$ times more likely to be chosen when the choice does not affect coverage.

Type:           Primary (10), Logical, Single, Span, Stripe, Mirror, RAID-5
Format method:  quick, slow
File system:    FAT, FAT32, NTFS (10)

PICT uses a simple type system to evaluate constraints. Types are inferred automatically:

• Numeric Type: A parameter is considered numeric if all its values can be converted to an integer or a float. Numeric parameters can be compared using >, <, >=, <=, etc.
• String Type: All other parameters are treated as strings. String parameters are compared using =, <>, or LIKE.

Note: If a value has multiple names, PICT uses only the first name to determine the type.

Aliasing allows you to provide multiple names for a single value. This does not increase combinatorial complexity; PICT treats them as one entity and rotates the names among test cases in the output.

Configuration:

• By default, names are separated by the | character.
• Use the /a option to change the alias separator.

Important Constraints Note: When evaluating constraints, only the first name counts. For example, if SKU_1 is aliased as Server | Datacenter, the constraint [SKU_1] = "Server" will match, but [SKU_1] = "Datacenter" will not. Similarly, only the first name is used for type detection or determining if a value is "negative" (out-of-range).

OS_1:   Win2008, Win2012, Win2016
SKU_1:  Professional, Server | Datacenter

PICT runs as a command-line tool. You provide it with a model file describing your parameters and their possible values.

Example model structure:

Type:          Single, Span, Stripe, Mirror, RAID-5
Size:          10, 100, 500, 1000, 5000, 10000, 40000
Format method: Quick, Slow
File system:   FAT, FAT32, NTFS
Cluster size:  512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
Compression:   On, Off

Download the latest pict.exe from the official releases page.

PICT uses CMake for building on Linux-based systems. Assuming you have CMake and a C++ toolchain installed, follow these steps:

cmake -DCMAKE_BUILD_TYPE=Release -S . -B build
cmake --build build
pushd build && ctest -V && popd

To build PICT on Windows, use the provided Visual Studio solution or the command-line script.

Using Visual Studio

1. Open pict.sln in Visual Studio 2022.
2. Ensure VC++ build tools are installed.

Using Command Line

Run the _build.cmd script in the root directory to build both Debug and Release configurations.

Running Tests

1. Ensure Perl is installed.
2. Run the _test.cmd script.
3. Check the generated logs (dbg.log for Debug or rel.log for Release) against committed baselines.

Negative testing involves using values outside the allowable range to ensure error handling. To prevent "input masking" (where one invalid input prevents another from being tested), PICT ensures that out-of-range values are paired only with valid values.

Usage:

• Prefix an invalid value with ~ (tilde) to mark it as out-of-range.
• Use the /n option to specify a different prefix if desired.

Behavior:

• The ~ prefix is not part of the value during constraint evaluation or type detection. For example, if A has value ~-1, the constraint if [A] = -1 then ... will work correctly.
• The ~ prefix will appear in the final output.
• If a value has multiple names (aliases), only prefixing the first name makes the entire value out-of-range.

A: ~-1, 0, 1, 2
B: ~-1, 0, 1, 2

PICT is a command-line tool that generates test cases from a plain-text model file. The basic usage pattern is:

pict model [options]

By default, PICT generates a pairwise test suite (covering all pairs of values). You can increase the combination order using the /o option (e.g., /o:3 for triplets). The output is printed to standard output as a tab-separated value format, where the first line contains parameter names and subsequent lines contain the generated test cases.