Random Testing

So, I had a need the other day to test various permutations of some numbers (and ignoring what this is or why for the moment). Being too lazy to think about coding the permutations, I just used Collections#shuffle(List<?>). In my case, it got the job done:

import static java.lang.Math.*;
import static org.junit.Assert.*;

import java.util.*;

import org.junit.*;

public class Whatever {

    private int secondGreatest(int a, int b, int c, int d) {
        return min(max(a, b), max(c, d));
    }

    @Test
    public void testSecondGreatest() {
        List numbers = Arrays.asList(1, 2, 3, 4);
        for (int i = 0; i < 20; i++) {
            Collections.shuffle(numbers);
            assertEquals(
                "for list " + numbers,
                3,
                secondGreatest(
                    numbers.get(0),
                    numbers.get(1),
                    numbers.get(2),
                    numbers.get(3)
                )
            );
        }
    }

}

It fails, for instance, with text like "java.lang.AssertionError: for list [4, 3, 2, 1] expected:<3> but was:<2>", but it's the not the same failure every time, and there's a probability (which I haven't calculated) that it might pass even though it shouldn't.

So, what's the big picture question? Lots of test data might be a bother to produce in advance or even impossible if the domain is huge. Even an algorithm to test every possibility might take too long. Or I might just be lazy like in this case. Or, in messier contexts, it might be impossible to guarantee the results perfectly, so the results themselves (not just the test cases) might be random. So you have to deal with that in some fashion.

Summary, seems like random (or "stochastic" if I want to sound academic?) testing can be either convenient or necessary in some circumstances, but it doesn't necessarily prove that the program works as expected. Might beat no testing in many cases.