Thursday, October 16, 2008

Random Testing

So, I had a need the other day to test various permutations of some numbers (and ignoring what this is or why for the moment). Being too lazy to think about coding the permutations, I just used Collections#shuffle(List<?>). In my case, it got the job done:

import static java.lang.Math.*;
import static org.junit.Assert.*;

import java.util.*;

import org.junit.*;

public class Whatever {

private int secondGreatest(int a, int b, int c, int d) {
return min(max(a, b), max(c, d));
}

@Test
public void testSecondGreatest() {
List numbers = Arrays.asList(1, 2, 3, 4);
for (int i = 0; i < 20; i++) {
Collections.shuffle(numbers);
assertEquals(
"for list " + numbers,
3,
secondGreatest(
numbers.get(0),
numbers.get(1),
numbers.get(2),
numbers.get(3)
)
);
}
}

}


It fails, for instance, with text like "java.lang.AssertionError: for list [4, 3, 2, 1] expected:<3> but was:<2>", but it's the not the same failure every time, and there's a probability (which I haven't calculated) that it might pass even though it shouldn't.

So, what's the big picture question? Lots of test data might be a bother to produce in advance or even impossible if the domain is huge. Even an algorithm to test every possibility might take too long. Or I might just be lazy like in this case. Or, in messier contexts, it might be impossible to guarantee the results perfectly, so the results themselves (not just the test cases) might be random. So you have to deal with that in some fashion.

Summary, seems like random (or "stochastic" if I want to sound academic?) testing can be either convenient or necessary in some circumstances, but it doesn't necessarily prove that the program works as expected. Might beat no testing in many cases.

2 comments:

  1. Randomly-generated test cases can be very convenient at times. They can even allow a shift from test case based unit testing to a specification-based approach.

    If you like randomly-generated test cases, you should use a dedicated framework rather than ad-hoc methods like Collections.shuffle.

    I think QuickCheck was one of the first practical framework for doing that. It is for the functional language Haskell, but has then be ported to other imperative languages (e.g. QuickCheck for Java, QuickCheck for C++).

    ReplyDelete