Ways to Test

Unit Tests

The first way is the most obvious. Write out a test case and verify it makes sense.

We could write that 2 + 2 should be 4.

#[test]
fn unit() {
    assert_eq!(add(2, 2), 4);
}

But there are a few problems:

1. We have to know what inputs to provide the function and what output that would return before writing the test case.

Let’s say I misunderstand how addition works, and write this implementation:

fn add(a: u32, b: u32) -> u32 {
    a + b + 1
}

It’s pretty easy to also write my test program incorrectly:

#[test]
fn unit() {
    assert_eq!(add(2, 2), 5);
}

And then the unit test is worse than useless – it reassures me that I’m testing an input properly when I am not.

2. There are 4 billion choices for the first parameter to the function, and 4 billion more choices for the second. I can’t write all of these test cases by hand. Moreover, it’s hard to find interesting test cases that actually trigger a bug.

Snapshot Tests

Snapshot tests are like unit tests, but instead of having to write out the output, you verify that the output is correct. This is useful for cases where it’s a pain to write the output value, like for an AST in a compiler.

They make it easier to write tests, but they still share the pitfall with unit tests, where you still need to know what went wrong.

Mutation Tests

Mutation tests are a bit different from the previous ways of testing: they take your existing unit tests and change (mutate) the body of a function under test, and check if the tests still hold.

For example, for the add function, it might change the operator or the numbers to something else like so:

fn add(a: u32, b: u32) -> u32 {
    a - b
}

The test runner would then run all of your unit tests and note whether or not they fail. If they don’t all fail, then there’s some overlap between this new program and your program, and either the tests are not representative enough, or the function could be implemented incorrectly.

Either way, mutation tests are useful because even if you implement a function incorrectly, they have a chance of telling you that you’ve done something incorrectly.

But they do give false positives:

Imagine the add function is mutated to the following:

fn add(a: u32, b: u32) -> u32 {
    a * b
}

And the test case is:

#[test]
fn unit() {
    assert_eq!(add(2, 2), 4);
}

This test case passes with the unchanged add function and the newly changed one. Thus, this function will be flagged as potentially implemented incorrectly, even if it is correct.

Fuzzing

What happens if we want to try to generate some use cases? One way of doing this is by using some “interesting” cases and see what happens.

If we do the following:

add(u32::MAX, u32::MAX);

We’ll get an integer overflow, and the program will crash.

Fuzzing creates interesting test cases and then tests to see if the function(s) under test crash with some input. If it does, it tries to find the shortest input that does so.

However, it’s not foolproof:

Imagine we wrote add like so:

fn add(a: u32, b: u32) -> u32 {
    a.saturating_add(b)
}

This function should no longer crash, rendering our fuzz tests useless.

Fuzzing is good for testing lots of use cases, but we lose a nice property – we’re just trying to make the function crash, and no longer verifying its output.

Property Tests

What if we know some things about the function we’re trying to test?

If we add two numbers where both numbers are >= 0, we know that the number has to be at least as large as the maximum of both numbers.

We can formalize this:

#[test]
fn property() {
    let X: u32 = random_number();
    let Y: u32 = random_number();
    let result = add(X, Y);
    assert!(result >= X.max(Y));
}

We have now found some useful property to test, so instead of just testing for a crash, we also have some guarantee that the function is correct.

That being said, we could still fool the property by writing this:

fn add(a: u32, b: u32) -> u32 {
    a.max(b) + 1
}

Which would pass our previous property test.