Property Based Testing for Objective-C and Swift. Automatic generation of software tests.

You might have heard of this or similar technologies through the various genres of testing frameworks and/or libraries:

• Haskell QuickCheck
• Clojure test.check
• Erlang QuickCheck (most advanced but commerical)
• Property Based Testing
• Model Based Testing
• Simulation Testing

Fox is a port of test.check for Objective-C. Unlike some ports of QuickCheck, Fox does implement shrinking (test.check does implement that too).

More thorough than Example-Based Tests

Test generation can provide better coverage than example-based tests. Instead of having to manually code test cases, Fox can generate tests for you.

Data Generation

The simpliest of test generation is providing random data. Fox can generate them for use if you can define specifications -- known properties of the subject under test:

FOXAssert(FOXForAll(FOXTuple(FOXInteger(), FOXInteger()), ^BOOL(NSArray *values){
    NSInteger x = [tuple[0] integerValue];
    NSInteger y = [tuple[1] integerValue];
    return x + y > x;
});

Once a failing example is produced, Fox will attempt to find the smallest possible example that also exhibits the same failure:

Property failed with: @[@0, @0].

Stateful Testing

How can you test stateful APIs? Represent the state changes as data! Using a state machine, define a model of how your API is suppose to work. Here's one for a queue:

// define a state machine. Model state is the state of your application and
// can be represented with any object you want -- Fox does not interpret it.
FOXFiniteStateMachine *stateMachine = [[FOXFiniteStateMachine alloc] initWithInitialModelState:@[]];

// Adds a transition to the state machine:
// - The API to test is -[addObject:]
// - The generator for the argument is a random integer in an NSNumber
// - A block indicating how to update the model state. This should not mutate the original model state.
[stateMachine addTransition:[FOXTransition byCallingSelector:@selector(addObject:)
                                               withGenerator:FOXInteger()
                                              nextModelState:^id(NSArray *modelState, id generatedValue) {
    return [modelState arrayByAddingObject:generatedValue];
}]];
// Add a custom transition (see FOXStateTransition protocol)
[stateMachine addTransition:[[QueueRemoveTransition alloc] init]];

Now, you can generate tests that exercise an API:

// Generate a sequence of commands executed on the given subject. Since
// this will generate multiple tests, you also give a block of a subject.
id<FOXGenerator> executedCommands = FOXExecuteCommands(stateMachine, ^id {
    return [FOXQueue new];
});

// Verify if the executed commands validated the API conformed to the state machine.
FOXRunnerResult *result = [FOXSpecHelper resultForAll:executedCommands
                                                 then:^BOOL(NSArray *commands) {
    return FOXExecutedSuccessfully(commands);
}];
// result will shrinking to the small sequence of API calls to trigger the
// failure if there is one

Read more at the latest documentation, or the stable documentation.

Installation

From the documentation.

Fox can be installed in multiple ways. If you don't have a preference, install via git submodule.

Fox honors semantic versioning as humanly possible. If you're unsure if a given update is backwards incompatible with your usage. Check out the releases.

Manually (Git Submodule)

Add Fox as a submodule to your project:

$ git submodule add https://github.com/jeffh/Fox.git Externals/Fox

If you don't want bleeding edge, check out the particular tag of the version:

$ cd Externals/Fox
$ git checkout v1.0.1

Add Fox.xcodeproj to your Xcode project (not Fox.xcworkspace). Then link Fox-iOS or Fox-OSX to your test target.

And you're all set up! Dive right in by following the tutorial.

Reference

If you want to see examples of usages, see the full reference.

Data Generators

There are many data generators provided for generating data. Most of these generators shrink to zero:

• Numerically zero (or as close as possible)
• Empty collection (or at least shrunk items)

Computation Generators

Also, you can compose some computation work on top of data generators. The resulting generator adopts the same shrinking properties as the original generator.